[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 531774484 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2496MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002401] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006809] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009143] LSM: Security Framework initializing [ 0.010058] Yama: becoming mindful. [ 0.011046] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.021007] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027092] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030076] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] *** VALIDATE tmpfs *** [ 0.033280] *** VALIDATE proc *** [ 0.034267] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037245] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039197] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041032] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045479] debug: unmapping init [mem 0xffffffffb3e59000-0xffffffffb3e60fff] [ 0.047899] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049028] ... version: 2 [ 0.050012] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052018] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054011] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.057233] rcu: Hierarchical SRCU implementation. [ 0.059446] smp: Bringing up secondary CPUs ... [ 0.060615] x86: Booting SMP configuration: [ 0.061023] .... node #0, CPUs: #1 #2 #3 [ 0.065101] smp: Brought up 1 node, 4 CPUs [ 0.067016] smpboot: Max logical packages: 1 [ 0.068014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144495] node 0 deferred pages initialised in 73ms [ 0.148469] devtmpfs: initialized [ 0.150356] x86/mm: Memory block size: 128MB [ 0.155144] gcov: version magic: 0x41383552 [ 0.156608] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163379] pinctrl core: initialized pinctrl subsystem [ 0.166221] [ 0.166825] ************************************************************* [ 0.170016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173013] ** ** [ 0.176013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178010] ** ** [ 0.181014] ** This means that this kernel is built to expose internal ** [ 0.183014] ** IOMMU data structures, which may compromise security on ** [ 0.186012] ** your system. ** [ 0.188013] ** ** [ 0.191023] ** If you see this message and you are not debugging the ** [ 0.194014] ** kernel, report this immediately to your vendor! ** [ 0.197014] ** ** [ 0.199012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202014] ************************************************************* [ 0.205853] NET: Registered protocol family 16 [ 0.208783] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.212081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.216092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222064] cpuidle: using governor menu [ 0.223785] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.227617] PCI: Using configuration type 1 for base access [ 0.229169] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.241209] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.244050] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.249063] cryptd: max_cpu_qlen set to 1000 [ 0.252313] ACPI: Added _OSI(Module Device) [ 0.254023] ACPI: Added _OSI(Processor Device) [ 0.256055] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.258023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.264359] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.272333] ACPI: Interpreter enabled [ 0.274079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.276018] ACPI: Using IOAPIC for interrupt routing [ 0.277127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.281379] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.292000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.296046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.299021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.303095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.308493] acpiphp: Slot [2] registered [ 0.310167] acpiphp: Slot [5] registered [ 0.312170] acpiphp: Slot [6] registered [ 0.313131] acpiphp: Slot [7] registered [ 0.316201] acpiphp: Slot [8] registered [ 0.317202] acpiphp: Slot [9] registered [ 0.319185] acpiphp: Slot [10] registered [ 0.321189] acpiphp: Slot [3] registered [ 0.323128] acpiphp: Slot [4] registered [ 0.325406] acpiphp: Slot [11] registered [ 0.328180] acpiphp: Slot [12] registered [ 0.330133] acpiphp: Slot [13] registered [ 0.332132] acpiphp: Slot [14] registered [ 0.334115] acpiphp: Slot [15] registered [ 0.337157] acpiphp: Slot [16] registered [ 0.339110] acpiphp: Slot [17] registered [ 0.340104] acpiphp: Slot [18] registered [ 0.342145] acpiphp: Slot [19] registered [ 0.344120] acpiphp: Slot [20] registered [ 0.345157] acpiphp: Slot [21] registered [ 0.347125] acpiphp: Slot [22] registered [ 0.349157] acpiphp: Slot [23] registered [ 0.350138] acpiphp: Slot [24] registered [ 0.352169] acpiphp: Slot [25] registered [ 0.354131] acpiphp: Slot [26] registered [ 0.355129] acpiphp: Slot [27] registered [ 0.357139] acpiphp: Slot [28] registered [ 0.358119] acpiphp: Slot [29] registered [ 0.359114] acpiphp: Slot [30] registered [ 0.361094] acpiphp: Slot [31] registered [ 0.362058] PCI host bridge to bus 0000:00 [ 0.364018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.366023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.368018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.370025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.373040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.376046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.378215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.381349] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.385639] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.394024] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.401063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.404022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.406018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.409019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.412536] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.415790] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.418035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.421944] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.427017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.440017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.446014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.451461] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.462029] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.470027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.498056] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.505016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.514016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.537020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.548000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.555021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.562023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.585015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.596181] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.607025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.614024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.630024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.640328] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.648018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.655016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.679013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.691899] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.699022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.708015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.725016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.738905] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.743399] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.746348] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.749460] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.751558] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.760090] iommu: Default domain type: Passthrough [ 0.762442] SCSI subsystem initialized [ 0.764214] ACPI: bus type USB registered [ 0.766092] usbcore: registered new interface driver usbfs [ 0.768071] usbcore: registered new interface driver hub [ 0.770079] usbcore: registered new device driver usb [ 0.771120] pps_core: LinuxPPS API ver. 1 registered [ 0.772007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.775058] PTP clock support registered [ 0.777120] EDAC MC: Ver: 3.0.0 [ 0.779167] PCI: Using ACPI for IRQ routing [ 0.781981] NetLabel: Initializing [ 0.783013] NetLabel: domain hash size = 128 [ 0.785011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.787149] NetLabel: unlabeled traffic allowed by default [ 0.790089] vgaarb: loaded [ 0.791197] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.793013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.799373] clocksource: Switched to clocksource kvm-clock [ 0.909583] VFS: Disk quotas dquot_6.6.0 [ 0.911618] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.914398] *** VALIDATE ramfs *** [ 0.915707] *** VALIDATE hugetlbfs *** [ 0.917547] pnp: PnP ACPI init [ 0.920037] pnp: PnP ACPI: found 6 devices [ 0.936651] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.940516] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.943367] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.945819] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.948730] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.951314] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.954914] NET: Registered protocol family 2 [ 0.957737] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.962662] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.967385] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.973711] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.977474] TCP: Hash tables configured (established 65536 bind 65536) [ 0.980597] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.984197] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.987335] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991386] NET: Registered protocol family 1 [ 0.994066] RPC: Registered named UNIX socket transport module. [ 0.996348] RPC: Registered udp transport module. [ 0.998236] RPC: Registered tcp transport module. [ 1.000116] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.002962] NET: Registered protocol family 44 [ 1.004845] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.006889] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.008639] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.011234] PCI: CLS 0 bytes, default 64 [ 1.013166] Unpacking initramfs... [ 2.466280] debug: unmapping init [mem 0xffff9e0ffcc54000-0xffff9e0ffffbffff] [ 2.470156] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.472269] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.474760] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.975980] Initialise system trusted keyrings [ 2.977976] Key type blacklist registered [ 2.980247] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.994336] zbud: loaded [ 2.997618] *** VALIDATE nfs *** [ 2.999063] *** VALIDATE nfs4 *** [ 3.000913] pstore: using deflate compression [ 3.005028] Platform Keyring initialized [ 3.107080] NET: Registered protocol family 38 [ 3.109251] Key type asymmetric registered [ 3.110991] Asymmetric key parser 'x509' registered [ 3.115428] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.119600] io scheduler mq-deadline registered [ 3.121318] io scheduler kyber registered [ 3.123033] io scheduler bfq registered [ 3.125285] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.128517] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.133841] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.136938] ACPI: Power Button [PWRF] [ 3.142598] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.149850] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.168816] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.179832] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.200863] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.228727] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.256572] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.262136] Non-volatile memory driver v1.3 [ 3.264039] Linux agpgart interface v0.103 [ 3.295736] virtio_blk virtio1: [vda] 146640 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.298993] vda: detected capacity change from 0 to 75079680 [ 3.314923] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.318477] vdb: detected capacity change from 0 to 1073741824 [ 3.332948] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.336794] vdc: detected capacity change from 0 to 2621440000 [ 3.355477] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.358745] vdd: detected capacity change from 0 to 2621440000 [ 3.375151] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.379378] vde: detected capacity change from 0 to 4294967296 [ 3.397517] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.401108] vdf: detected capacity change from 0 to 4294967296 [ 3.407992] libphy: Fixed MDIO Bus: probed [ 3.414723] usbcore: registered new interface driver usbserial_generic [ 3.417325] usbserial: USB Serial support registered for generic [ 3.420329] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.424945] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.427130] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.429985] mousedev: PS/2 mouse device common for all mice [ 3.433285] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.434873] rtc_cmos 00:05: RTC can wake from S4 [ 3.440186] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.441192] rtc_cmos 00:05: registered as rtc0 [ 3.445688] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.446219] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.452869] intel_pstate: CPU model not supported [ 3.456219] hid: raw HID events driver (C) Jiri Kosina [ 3.458739] usbcore: registered new interface driver usbhid [ 3.461089] usbhid: USB HID core driver [ 3.462918] drop_monitor: Initializing network drop monitor service [ 3.465650] Initializing XFRM netlink socket [ 3.467541] NET: Registered protocol family 10 [ 3.470546] Segment Routing with IPv6 [ 3.472258] NET: Registered protocol family 17 [ 3.474797] mpls_gso: MPLS GSO support [ 3.480460] RAS: Correctable Errors collector initialized. [ 3.482478] AVX version of gcm_enc/dec engaged. [ 3.484732] AES CTR mode by8 optimization enabled [ 3.563656] sched_clock: Marking stable (3563603583, 0)->(4487542328, -923938745) [ 3.570637] registered taskstats version 1 [ 3.572473] Loading compiled-in X.509 certificates [ 3.574480] zswap: loaded using pool lzo/zbud [ 3.618600] Key type big_key registered [ 3.636876] Key type encrypted registered [ 3.638948] ima: No TPM chip found, activating TPM-bypass! [ 3.641737] ima: Allocated hash algorithm: sha1 [ 3.643960] ima: No architecture policies found [ 3.645758] evm: Initialising EVM extended attributes: [ 3.647442] evm: security.selinux [ 3.648694] evm: security.ima [ 3.649792] evm: security.capability [ 3.651040] evm: HMAC attrs: 0x1 [ 3.653854] rtc_cmos 00:05: setting system clock to 2026-08-27 13:50:11 UTC (1787838611) [ 3.660570] debug: unmapping init [mem 0xffffffffb4e03000-0xffffffffb4ffffff] [ 3.663761] debug: unmapping init [mem 0xffffffffb3b82000-0xffffffffb3e58fff] [ 3.675310] Write protecting the kernel read-only data: 28672k [ 3.679334] debug: unmapping init [mem 0xffffffffb2203000-0xffffffffb23fffff] [ 3.682849] debug: unmapping init [mem 0xffffffffb2b14000-0xffffffffb2bfffff] [ 3.727960] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.735858] systemd[1]: Detected virtualization kvm. [ 3.738091] systemd[1]: Detected architecture x86-64. [ 3.740323] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.764885] systemd[1]: No hostname configured. [ 3.766752] systemd[1]: Set hostname to . [ 3.769169] random: systemd: uninitialized urandom read (16 bytes read) [ 3.771768] systemd[1]: Initializing machine ID from random generator. [ 3.828834] random: ln: uninitialized urandom read (6 bytes read) [ 3.938530] random: systemd: uninitialized urandom read (16 bytes read) [ 3.941597] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.948538] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.954862] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.697208] device-mapper: uevent: version 1.0.3 [ 4.699416] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.485981] virtio_net virtio0 ens2: renamed from eth0 [ 5.514936] random: fast init done [ 5.520130] scsi host0: ata_piix [ 5.525325] scsi host1: ata_piix [ 5.528153] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.533123] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.219554] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.156031] random: crng init done [ 10.157809] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.670678] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.863448] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.154135] SELinux: Disabled at runtime. [ 12.213524] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.223386] systemd[1]: Detected virtualization kvm. [ 12.225218] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.751779] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.756514] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.761593] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.766255] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.770592] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.778503] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.784470] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ 12.926673] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.266442] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.589641] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.614715] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.704095] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.722974] EDAC sbridge: Ver: 1.1.2 [ 15.539741] Key type dns_resolver registered [ 15.852417] NFS: Registering the id_resolver key type [ 15.854350] Key type id_resolver registered [ 15.855903] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg427-server login: [ 46.626443] libcfs: loading out-of-tree module taints kernel. [ 46.711554] Key type ._llcrypt registered [ 46.713948] Key type .llcrypt registered [ 46.851596] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 67.653295] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 69.592905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 69.606470] alg: No test for adler32 (adler32-zlib) [ 71.024289] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 72.464649] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 74.394527] Key type lgssc registered [ 75.745851] hrtimer: interrupt took 4717541 ns [ 76.363527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.897284] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 145.217253] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 159.705863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.774800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.102616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.147366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.237835] Lustre: lustre-MDT0000: new disk, initializing [ 161.355684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.381699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 166.567687] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.825347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.915254] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 180.980401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 180.989396] Lustre: Skipped 1 previous similar message [ 181.079512] Lustre: lustre-MDT0001: new disk, initializing [ 181.156091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.184145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 181.203962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.606601] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.738779] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.735062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.076567] Lustre: lustre-OST0000: new disk, initializing [ 205.080296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.086831] Lustre: 8445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.148594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.767918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.781628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.845385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.058698] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.624668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.771539] Lustre: lustre-OST0001: new disk, initializing [ 229.776724] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.786925] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.863300] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.643282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 236.659880] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 236.750747] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 237.268732] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.389414] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 259.360986] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.912429] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing check_logdir /tmp/testlogs/ [ 272.098566] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing yml_node [ 276.114207] Lustre: DEBUG MARKER: Client: 2.17.57.85 [ 278.706084] Lustre: DEBUG MARKER: MDS: 2.17.57.85 [ 281.317229] Lustre: DEBUG MARKER: OSS: 2.17.57.85 [ 283.181333] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 09:54:49 EDT 2026 [ 301.545511] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 303.139689] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 324.065196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.068710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.085221] Lustre: Skipped 1 previous similar message [ 324.118720] Lustre: Skipped 3 previous similar messages [ 329.185694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.201132] Lustre: Skipped 3 previous similar messages [ 334.305733] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 334.306532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 334.329226] Lustre: Skipped 3 previous similar messages [ 334.750144] Lustre: server umount lustre-MDT0000 complete [ 338.971450] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838946 with bad export cookie 1775379396954459371 [ 338.977654] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.997844] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 339.425044] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.435714] Lustre: Skipped 2 previous similar messages [ 339.441246] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.445830] Lustre: Skipped 1 previous similar message [ 343.470590] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.298572] Lustre: server umount lustre-MDT0001 complete [ 349.613644] Lustre: server umount lustre-OST0000 complete [ 353.331033] Lustre: server umount lustre-OST0001 complete [ 361.819988] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 371.854424] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 409.842806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.033558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 410.066901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 410.188875] Lustre: lustre-MDT0000: new disk, initializing [ 410.263631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.282143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 414.979575] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 427.410655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.569175] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 427.636432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 427.643206] Lustre: Skipped 1 previous similar message [ 427.703891] Lustre: lustre-MDT0001: new disk, initializing [ 427.780628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 427.813885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 427.826661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 434.295735] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 448.559205] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.145507] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 462.608936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.859473] Lustre: lustre-OST0000: new disk, initializing [ 462.866817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.879779] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.964657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.735722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.746418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.916449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.373829] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 482.445839] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 488.996611] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 489.380227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 496.261961] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 496.558672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 498.661181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 498.668318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 498.681784] Lustre: Skipped 2 previous similar messages [ 498.705943] Lustre: Skipped 2 previous similar messages [ 504.546931] Lustre: server umount lustre-OST0000 complete [ 508.897144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.897144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.140400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.150154] Lustre: Skipped 7 previous similar messages [ 523.232158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 523.400206] Lustre: server umount lustre-MDT0000 complete [ 524.257409] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.305397] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 527.274189] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839135 with bad export cookie 1775379396954460904 [ 527.278453] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.288699] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 527.565965] Lustre: server umount lustre-MDT0001 complete [ 533.545663] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 09:58:59 (1787839139) [ 541.533506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.379130] LustreError: 19309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.458994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.657838] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 547.815093] LustreError: 19310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.935692] LustreError: 19309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.068310] LustreError: 19310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.901245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.695762] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 572.479441] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 579.169575] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 587.165523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.397337] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 602.384885] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 617.331741] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 626.146537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 626.160288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.179080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 631.276938] Lustre: server umount lustre-MDT0000 complete [ 633.830464] LustreError: 20037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.856643] LustreError: 20037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 635.325075] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839243 with bad export cookie 1775379396954461786 [ 635.330171] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.340304] LustreError: 19289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 635.690549] Lustre: server umount lustre-MDT0001 complete [ 649.775626] Lustre: server umount lustre-OST0000 complete [ 675.064620] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 686.934931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.568154] LustreError: 23729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.700496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 687.709739] Lustre: Skipped 2 previous similar messages [ 693.267963] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 702.711379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.121616] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 711.358676] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 719.150521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 727.940363] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 739.102311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.283415] Lustre: lustre-OST0001: new disk, initializing [ 739.286557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 739.294634] Lustre: 26047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 740.617956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 740.651946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 740.724487] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 747.699996] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 757.854575] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 761.975480] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 774.642367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 774.658373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.681703] Lustre: Skipped 2 previous similar messages [ 774.691806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 774.707710] Lustre: Skipped 3 previous similar messages [ 780.597593] Lustre: server umount lustre-MDT0000 complete [ 781.796934] LustreError: 23730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 781.818960] LustreError: 23730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 784.575200] LustreError: 23710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839392 with bad export cookie 1775379396954462633 [ 784.581173] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.584747] LustreError: 23710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 785.051954] Lustre: server umount lustre-MDT0001 complete [ 800.015848] Lustre: server umount lustre-OST0000 complete [ 802.784148] Lustre: 3640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839394/real 1787839394] req@ffff9e10574ac380 x1874684731977984/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787839410 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 805.675645] Lustre: server umount lustre-OST0001 complete [ 828.645321] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 840.490295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.115496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 841.127454] Lustre: Skipped 3 previous similar messages [ 846.309874] LustreError: 29064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.330318] LustreError: 29064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 846.540605] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 856.887393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.407462] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 866.341213] Lustre: 30215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 875.012024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 882.799797] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 892.641898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.709934] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 908.064830] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 911.990439] Lustre: 32096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 919.017781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.025149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.028100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 919.054286] Lustre: Skipped 6 previous similar messages [ 919.063941] Lustre: Skipped 5 previous similar messages [ 925.329243] Lustre: server umount lustre-MDT0000 complete [ 929.663618] LustreError: 29046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839537 with bad export cookie 1775379396954463928 [ 929.674032] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 929.694039] LustreError: 29046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 930.555945] Lustre: server umount lustre-MDT0001 complete [ 946.217256] Lustre: server umount lustre-OST0000 complete [ 950.745185] Lustre: 3639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839542/real 1787839542] req@ffff9e107ee7f800 x1874684732030976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787839558 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 951.128817] Lustre: server umount lustre-OST0001 complete [ 961.515788] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 10:06:07 (1787839567) [ 992.977656] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1005.167541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.563004] LustreError: 34901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1005.592991] LustreError: 34901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 1010.879108] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1021.638521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.990850] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1031.830695] Lustre: 36051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1040.271713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1048.088403] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1059.105972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1068.344083] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1077.520473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1081.930475] Lustre: 37933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1088.997249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1089.011954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.026139] Lustre: Skipped 2 previous similar messages [ 1094.476132] Lustre: server umount lustre-MDT0000 complete [ 1099.119225] LustreError: 34883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839706 with bad export cookie 1775379396954465034 [ 1099.122865] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.134944] LustreError: 34883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1099.695590] Lustre: server umount lustre-MDT0001 complete [ 1115.356577] Lustre: server umount lustre-OST0000 complete [ 1116.513753] Lustre: 3641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839708/real 1787839708] req@ffff9e105b469180 x1874684732087552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787839724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1119.988885] Lustre: server umount lustre-OST0001 complete [ 1130.336222] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 10:08:56 (1787839736) [ 1148.986629] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1160.080412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.598107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.616622] Lustre: Skipped 7 previous similar messages [ 1165.364910] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1174.726505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.153892] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1184.667255] Lustre: 41223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1193.050431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1200.160316] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1210.066407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1217.662947] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1225.267798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1229.142584] Lustre: 43099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1230.052509] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1230.060202] Lustre: Skipped 1 previous similar message [ 1235.827509] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 1235.846687] Lustre: Skipped 11 previous similar messages [ 1235.939155] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.967954] Lustre: Skipped 6 previous similar messages [ 1239.732612] Lustre: server umount lustre-MDT0000 complete [ 1243.434081] LustreError: 43101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839851 with bad export cookie 1775379396954466140 [ 1243.453911] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1243.457540] LustreError: 43101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1249.816254] Lustre: server umount lustre-MDT0001 complete [ 1258.461448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.706594] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e0f42dda300 x1874684732140800/t0(0) o250->MGC192.168.204.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1269.005070] LustreError: 41587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.029652] LustreError: 41587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 1273.518829] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1278.384599] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1281.875872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.684372] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1287.671481] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1287.679429] Lustre: Skipped 1 previous similar message [ 1294.472569] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1296.056581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1302.807615] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1305.290253] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1313.258637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1318.095902] Lustre: server umount lustre-MDT0000 complete [ 1321.801859] LustreError: 43101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839929 with bad export cookie 1775379396954467267 [ 1321.803615] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1321.818663] LustreError: 43101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1322.278491] Lustre: server umount lustre-MDT0001 complete [ 1337.340289] Lustre: server umount lustre-OST0000 complete [ 1339.748287] Lustre: 3639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839931/real 1787839931] req@ffff9e10574ac000 x1874684732176512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787839947 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1341.433597] Lustre: server umount lustre-OST0001 complete [ 1350.343857] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 10:12:36 (1787839956) [ 1368.123729] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1380.092354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.428330] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1397.339270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.013546] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1409.159986] Lustre: 48868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1419.216412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1428.824759] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1438.925679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1447.173754] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1456.857473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1462.031934] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1480.167435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1484.234946] Lustre: server umount lustre-MDT0000 complete [ 1488.476899] LustreError: 47695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840096 with bad export cookie 1775379396954468065 [ 1488.478429] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1488.486741] LustreError: 47695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1489.013575] Lustre: server umount lustre-MDT0001 complete [ 1502.420242] Lustre: server umount lustre-OST0000 complete [ 1517.279804] Lustre: server umount lustre-OST0001 complete [ 1529.498434] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 10:15:35 (1787840135) [ 1532.584523] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1534.550557] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 10:15:40 (1787840140) [ 1546.341310] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 1556.994539] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1582.917207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.194713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1583.265574] Lustre: lustre-MDT0000: new disk, initializing [ 1583.369555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1588.743171] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1599.590450] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1607.387231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1607.683650] Lustre: lustre-OST0000: new disk, initializing [ 1607.687705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1607.692620] Lustre: Skipped 1 previous similar message [ 1607.696190] Lustre: 54938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1609.477785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1609.492395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1609.534492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1615.082314] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1626.384923] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1628.757350] Lustre: DEBUG MARKER: create test files [ 1666.017459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1666.026312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.041326] Lustre: Skipped 12 previous similar messages [ 1671.167226] Lustre: server umount lustre-OST0000 complete [ 1675.578735] Lustre: server umount lustre-MDT0000 complete [ 1686.750540] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1689.578539] Key type lgssc unregistered [ 1689.898522] LNet: 56582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.907932] LNetError: 56582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.983733] LNet: Removed LNI 192.168.204.127@tcp [ 1691.983348] Key type .llcrypt unregistered [ 1691.986556] Key type ._llcrypt unregistered [ 1723.075667] Key type ._llcrypt registered [ 1723.078059] Key type .llcrypt registered [ 1723.200631] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1724.685915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1724.763891] alg: No test for adler32 (adler32-zlib) [ 1725.820614] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 1726.167842] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1727.848319] Key type lgssc registered [ 1728.859976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.185401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1736.208513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.781365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.650799] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1751.792539] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1759.740132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1760.156117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.222504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1762.289637] Lustre: 59310:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1762.304098] Lustre: 59310:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1762.313035] Lustre: 59310:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1762.328803] Lustre: 59310:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1762.345847] Lustre: 59310:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1762.351958] Lustre: 59310:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1765.877573] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1773.950811] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1779.704814] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1786.903803] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1796.571941] Lustre: server umount lustre-OST0000 complete [ 1810.516924] Lustre: server umount lustre-MDT0000 complete [ 1821.624550] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1824.638934] Key type lgssc unregistered [ 1825.025026] LNet: 60920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.044971] LNetError: 60920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.069281] LNet: Removed LNI 192.168.204.127@tcp [ 1825.872081] Key type .llcrypt unregistered [ 1825.874968] Key type ._llcrypt unregistered [ 1863.716690] Key type ._llcrypt registered [ 1863.719270] Key type .llcrypt registered [ 1863.838837] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1865.270106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.329911] alg: No test for adler32 (adler32-zlib) [ 1866.374716] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 1866.632424] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 1868.344540] Key type lgssc registered [ 1869.523508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1878.326143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1878.362835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.866396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.487520] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1891.898473] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1898.681969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1899.047629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.175229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1904.271682] Lustre: 63755:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1904.281772] Lustre: 63755:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1904.287669] Lustre: 63755:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1904.299092] Lustre: 63755:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1904.306976] Lustre: 63755:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1904.316488] Lustre: 63755:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1905.092101] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 1913.671501] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1919.008813] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1929.699391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.721144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.821069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1935.797293] Lustre: server umount lustre-OST0000 complete [ 1940.285652] Lustre: server umount lustre-MDT0000 complete [ 1954.235118] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 1957.966925] Key type lgssc unregistered [ 1958.326913] LNet: 65313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.333347] LNetError: 65313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.363251] LNet: Removed LNI 192.168.204.127@tcp [ 1959.314711] Key type .llcrypt unregistered [ 1959.323291] Key type ._llcrypt unregistered [ 1979.672641] Key type ._llcrypt registered [ 1979.678556] Key type .llcrypt registered [ 1979.808448] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 1998.400508] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 1999.871452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1999.902816] alg: No test for adler32 (adler32-zlib) [ 2000.973894] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 2001.237844] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 2002.960255] Key type lgssc registered [ 2004.335981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2039.081312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2039.122770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2040.473702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2040.507663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2040.615729] Lustre: lustre-MDT0000: new disk, initializing [ 2040.720888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.742627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2045.443803] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2059.235074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.359452] Lustre: 68463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2059.430536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2059.437397] Lustre: Skipped 1 previous similar message [ 2059.532366] Lustre: lustre-MDT0001: new disk, initializing [ 2059.642307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2059.715793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2059.734244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2064.497433] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2074.481630] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2080.523923] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2087.855575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2088.105785] Lustre: lustre-OST0000: new disk, initializing [ 2088.109201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2088.115055] Lustre: 70433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2088.203278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2093.625480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2093.632456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2093.754290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2094.389334] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2104.602652] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2108.983597] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2109.201477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2115.017221] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2115.450447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2119.657475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2119.659793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.672315] LustreError: Skipped 1 previous similar message [ 2119.720563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2119.734499] Lustre: Skipped 1 previous similar message [ 2124.145661] Lustre: server umount lustre-OST0000 complete [ 2129.890827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.903822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.918130] Lustre: Skipped 2 previous similar messages [ 2129.935755] Lustre: Skipped 1 previous similar message [ 2135.057914] Lustre: server umount lustre-MDT0000 complete [ 2140.137617] LustreError: 68471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.153632] LustreError: 68471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2140.217672] LustreError: 68456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840748 with bad export cookie 18092374230778994277 [ 2140.222065] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2140.228762] LustreError: 68456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2140.702392] Lustre: server umount lustre-MDT0001 complete [ 2153.325636] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 10:25:59 (1787840759) [ 2164.852425] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2214.041420] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2226.226077] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2269.177031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.497708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2269.558845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2269.716728] Lustre: lustre-MDT0000: new disk, initializing [ 2269.858703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.882707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2274.963169] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2287.530082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.669893] Lustre: 76457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2287.751787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2287.768364] Lustre: Skipped 1 previous similar message [ 2287.884220] Lustre: lustre-MDT0001: new disk, initializing [ 2287.982641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2288.039133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2288.076222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2293.640973] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2305.840745] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2313.861353] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2323.215582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2323.776777] Lustre: lustre-OST0000: new disk, initializing [ 2323.785085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2323.790611] Lustre: 78428:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2323.914470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2331.471839] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2331.729884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2331.753461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2331.947978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2343.367287] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2348.877430] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2349.132710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2355.216604] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2355.572391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2362.855372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2362.876151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.876797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.900670] Lustre: Skipped 1 previous similar message [ 2362.949620] Lustre: Skipped 1 previous similar message [ 2364.763534] Lustre: server umount lustre-OST0000 complete [ 2370.017318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2370.026866] LustreError: Skipped 1 previous similar message [ 2370.045859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.061522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.210184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.225954] Lustre: Skipped 3 previous similar messages [ 2384.352324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2384.610134] Lustre: server umount lustre-MDT0000 complete [ 2387.429181] LustreError: 76466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.452499] LustreError: 76466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2388.851950] LustreError: 76451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840996 with bad export cookie 18092374230778995159 [ 2388.854036] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2388.862719] LustreError: 76451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2389.298422] Lustre: server umount lustre-MDT0001 complete [ 2400.509733] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 10:30:06 (1787841006) [ 2408.009459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2408.677591] LustreError: 80842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.792287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.388463] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2414.049495] LustreError: 80843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.218324] LustreError: 80842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2423.015635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.266549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2423.277023] Lustre: Skipped 2 previous similar messages [ 2428.835418] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2438.412634] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2445.295066] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2455.427811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2455.926803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.935751] Lustre: Skipped 1 previous similar message [ 2464.668635] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2474.324834] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2478.670509] Lustre: 83515:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 7601a4b2-7841-47a4-abe4-63e97931a630 at adminstrative request [ 2483.686155] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.728787] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2493.948521] Lustre: MGS: Client 7601a4b2-7841-47a4-abe4-63e97931a630 (at 0@lo) reconnecting [ 2493.969879] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xfb15068161036549 to 0xfb150681610367e9 [ 2493.998780] Lustre: MGC192.168.204.127@tcp: Connection restored to 0@lo (at 0@lo) [ 2516.652773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2516.889360] Lustre: lustre-OST0001: new disk, initializing [ 2516.893938] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2516.900526] Lustre: 83907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2522.717183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2522.724045] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2522.870760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2524.074859] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2534.157303] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2538.486570] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2538.487233] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.491225] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2538.498426] LustreError: Skipped 1 previous similar message [ 2538.515413] Lustre: Skipped 2 previous similar messages [ 2543.588890] LustreError: 82664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.604266] LustreError: 82664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2543.679242] Lustre: server umount lustre-OST0001 complete [ 2548.710628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.729985] Lustre: Skipped 1 previous similar message [ 2554.855638] Lustre: server umount lustre-OST0000 complete [ 2561.508926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2574.306611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2574.317114] Lustre: Skipped 11 previous similar messages [ 2575.328810] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2575.835362] Lustre: server umount lustre-MDT0000 complete [ 2578.406634] LustreError: 80842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.434347] LustreError: 80842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2580.875351] LustreError: 80822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841188 with bad export cookie 18092374230778996713 [ 2580.876666] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2580.894039] LustreError: 80822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2581.304754] Lustre: server umount lustre-MDT0001 complete [ 2592.746552] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 10:33:18 (1787841198) [ 2594.671982] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2596.562917] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 10:33:22 (1787841202) [ 2598.010172] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2600.188522] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 10:33:26 (1787841206) [ 2603.668159] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2605.909645] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 10:33:31 (1787841211) [ 2607.521682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2609.587356] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 10:33:35 (1787841215) [ 2613.167319] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2623.009204] loop: module loaded [ 2623.046133] loop0: detected capacity change from 0 to 4294967296 [ 2623.133100] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2625.522479] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2629.944651] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2634.330975] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2665.977766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2666.033913] LustreError: 88040:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2666.044915] LustreError: 88040:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2666.062598] LustreError: 88040:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2666.081396] LustreError: 88040:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2689.635135] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 2700.689909] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 2745.174611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.453384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2745.476551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2745.549961] Lustre: lustre-MDT0000: new disk, initializing [ 2745.606098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.615125] Lustre: Skipped 1 previous similar message [ 2745.625161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2751.056045] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2765.066742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.245421] Lustre: 90905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2765.709840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2771.107484] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2781.806846] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2788.695134] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2796.650749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2796.967610] Lustre: 92877:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2803.290835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2803.445963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2804.869339] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2815.435505] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2821.240791] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2821.481755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2826.521635] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2826.747276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2829.283900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2829.298856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.311744] Lustre: Skipped 2 previous similar messages [ 2829.320228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.331756] Lustre: Skipped 1 previous similar message [ 2843.106627] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2843.436793] Lustre: server umount lustre-OST0000 complete [ 2854.344588] Lustre: server umount lustre-MDT0000 complete [ 2854.882691] LustreError: 92071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.930257] LustreError: 92071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2859.074331] LustreError: 92876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841466 with bad export cookie 18092374230778997329 [ 2859.082089] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2859.088515] LustreError: 92876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2859.908706] Lustre: server umount lustre-MDT0001 complete [ 2871.532336] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 10:37:57 (1787841477) [ 2879.941709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.626354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2880.629452] Lustre: Skipped 2 previous similar messages [ 2885.198661] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2894.206640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2899.028734] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2906.612582] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2912.462442] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2920.096265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2926.322948] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2937.068558] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2943.946273] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2944.348471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2949.508682] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2949.761497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2957.084717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2957.256571] Lustre: lustre-OST0001: new disk, initializing [ 2957.259459] Lustre: Skipped 2 previous similar messages [ 2957.263162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2957.270074] Lustre: Skipped 3 previous similar messages [ 2957.274381] Lustre: 98667:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2965.653639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2965.669524] Lustre: Skipped 2 previous similar messages [ 2965.688763] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2965.844666] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2966.113652] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 2977.800122] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2984.069484] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2984.285622] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2989.497290] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2989.747081] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3010.261940] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3040.036618] Lustre: Failing over lustre-MDT0000 [ 3040.729478] Lustre: server umount lustre-MDT0000 complete [ 3042.799560] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.809281] LustreError: 95291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.816145] Lustre: Skipped 6 previous similar messages [ 3042.841872] LustreError: 95291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3059.168533] Lustre: 66690:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841650/real 1787841650] req@00000000c8c0c192 x1874686754712192/t0(0) o400->MGC192.168.204.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787841666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3059.184920] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3066.901146] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3066.903407] LDISKFS-fs (dm-0): recovery complete [ 3066.925234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.697725] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3073.865496] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3073.868650] Lustre: 95291:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3074.213646] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3075.066253] Lustre: 95295:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3075.080300] Lustre: 95295:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 3075.119141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3075.234970] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3075.562293] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3075.896449] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3075.902379] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 3076.232209] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3076.242842] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 3076.263943] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3076.284984] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 3076.947974] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3076.981345] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 3077.320206] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3077.342202] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 3078.336345] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3078.359916] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 5 previous similar messages [ 3079.018239] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3079.037554] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 3079.360240] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3079.383040] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 3082.628071] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3082.645727] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 3083.293697] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3083.312151] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 3083.640160] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3083.646676] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 3090.682254] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3090.696930] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 3091.337163] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3091.355513] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 3091.672156] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3091.681657] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 3106.878254] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3106.906534] Lustre: 101518:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 3107.554377] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3107.569623] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 3107.888162] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3107.910200] LustreError: 101518:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 3133.032321] Lustre: 101518:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3133.054899] Lustre: 101518:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3133.076936] Lustre: 101518:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f156d8d0-4a35-4daf-b48c-01ec9ff19705@192.168.204.27@tcp [ 3133.092613] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3133.099053] Lustre: 101518:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@0000000097e9dcca x1874686734570880/t0(8589934777) o101->f156d8d0-4a35-4daf-b48c-01ec9ff19705@192.168.204.27@tcp:241/0 lens 592/0 e 0 to 0 dl 1787841751 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3133.147371] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3133.171063] Lustre: lustre-MDT0000: Denying connection for new client f156d8d0-4a35-4daf-b48c-01ec9ff19705 (at 192.168.204.27@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3133.210134] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3133.281050] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3133.288299] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3133.289056] Lustre: Skipped 2 previous similar messages [ 3133.360975] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 3133.362069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 3149.794390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3149.797296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.808450] LustreError: Skipped 2 previous similar messages [ 3149.809366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3149.809371] Lustre: Skipped 6 previous similar messages [ 3149.865969] Lustre: Skipped 1 previous similar message [ 3153.538442] Lustre: server umount lustre-OST0000 complete [ 3164.607623] Lustre: server umount lustre-OST0001 complete [ 3175.904385] Lustre: server umount lustre-MDT0000 complete [ 3177.448115] LustreError: 95295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.472775] LustreError: 95295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 3179.737173] LustreError: 95276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841787 with bad export cookie 18092374230779048548 [ 3179.755643] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3179.757526] LustreError: 95276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3180.301917] Lustre: server umount lustre-MDT0001 complete [ 3189.794573] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 10:43:15 (1787841795) [ 3197.631130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.635634] Lustre: *** cfs_fail_loc=197, val=0*** [ 3213.153805] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3218.273607] LustreError: 103490:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.127@tcp: failed processing log, type 4: rc = -110 [ 3223.411943] Lustre: *** cfs_fail_loc=197, val=0*** [ 3223.414905] Lustre: Skipped 65 previous similar messages [ 3244.064283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.082314] Lustre: Skipped 4 previous similar messages [ 3250.687252] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3259.414318] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3261.798342] Lustre: server umount lustre-OST0000 complete [ 3271.360758] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 10:44:37 (1787841877) [ 3292.941641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3297.933265] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3307.599430] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3312.292094] LustreError: 105138:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3312.299940] LustreError: 105138:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3312.305256] LustreError: 105138:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3312.959099] Lustre: server umount lustre-MDT0000 complete [ 3346.925821] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 3357.999448] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3398.507668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.822148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3398.898753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3399.053305] Lustre: lustre-MDT0000: new disk, initializing [ 3399.161644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3403.650868] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3416.289103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.392655] Lustre: 109314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3416.597524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3421.702745] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3432.784631] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3439.422821] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3446.954576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3447.202548] Lustre: 111283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3453.796192] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3456.694608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3463.686152] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3468.739326] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3468.921468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3473.472809] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3473.691884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3477.484963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3477.491458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3477.502195] Lustre: Skipped 4 previous similar messages [ 3477.505443] LustreError: Skipped 2 previous similar messages [ 3477.507821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3477.524347] Lustre: Skipped 10 previous similar messages [ 3481.834387] Lustre: server umount lustre-OST0000 complete [ 3491.561595] Lustre: server umount lustre-MDT0000 complete [ 3492.833554] LustreError: 110550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3492.851781] LustreError: 110550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3495.662176] LustreError: 109308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842103 with bad export cookie 18092374230779059195 [ 3495.667990] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3495.671789] LustreError: 109308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3496.139293] Lustre: server umount lustre-MDT0001 complete [ 3505.685460] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 10:48:31 (1787842111) [ 3515.274204] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 3518.362078] Key type lgssc unregistered [ 3518.702558] LNet: 113769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3518.709777] LNetError: 113769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3518.722478] LNet: Removed LNI 192.168.204.127@tcp [ 3519.580829] Key type .llcrypt unregistered [ 3519.585232] Key type ._llcrypt unregistered [ 3537.739983] Key type ._llcrypt registered [ 3537.743939] Key type .llcrypt registered [ 3537.872148] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 3554.590834] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3555.539375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3555.619645] alg: No test for adler32 (adler32-zlib) [ 3556.681670] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 3557.016491] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 3558.704186] Key type lgssc registered [ 3560.340278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3604.981765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3605.021417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.277782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3606.317896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3606.402165] Lustre: lustre-MDT0000: new disk, initializing [ 3606.493604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.520315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3611.078400] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3623.133329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3623.414376] Lustre: lustre-OST0000: new disk, initializing [ 3623.421237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3623.430617] Lustre: Skipped 1 previous similar message [ 3623.436352] Lustre: 117939:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3623.534820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3624.660627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3624.671370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3624.716297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3629.841774] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3643.875415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3644.111820] Lustre: lustre-OST0001: new disk, initializing [ 3644.116085] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3644.121300] Lustre: 118802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3644.205420] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3651.261852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3651.271990] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3651.324766] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3652.306407] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3661.796140] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.825279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.830421] Lustre: Skipped 1 previous similar message [ 3666.916019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3666.942759] Lustre: Skipped 1 previous similar message [ 3668.160355] Lustre: server umount lustre-MDT0000 complete [ 3675.556172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3681.145343] LustreError: 117128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842288 with bad export cookie 1466287636015773981 [ 3681.164487] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3681.256544] Lustre: server umount lustre-OST0000 complete [ 3685.534207] Lustre: server umount lustre-OST0001 complete [ 3695.136000] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 10:51:41 (1787842301) [ 3721.595975] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 10:52:07 (1787842327) [ 3733.996506] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3757.562633] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 10:52:43 (1787842363) [ 3797.481480] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 3807.190728] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 3846.632572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.027650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3847.074656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3847.224530] Lustre: lustre-MDT0000: new disk, initializing [ 3847.366965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3847.398420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3852.163948] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3863.784193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.911847] Lustre: 126561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3863.962664] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3863.966180] Lustre: Skipped 1 previous similar message [ 3864.062425] Lustre: lustre-MDT0001: new disk, initializing [ 3864.290966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3864.334322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3864.350689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3868.796420] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3879.765680] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3887.025494] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3894.203135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3894.611609] Lustre: lustre-OST0000: new disk, initializing [ 3894.620384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3894.631796] Lustre: 128527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3894.749319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.314633] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3902.535612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3902.549964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3902.646435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3911.735906] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3917.610536] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3917.860482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3923.558976] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3923.886941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3928.548720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.568738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3928.569411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3928.577741] Lustre: Skipped 2 previous similar messages [ 3932.445484] Lustre: server umount lustre-OST0000 complete [ 3938.790071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.798473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3938.818267] Lustre: Skipped 1 previous similar message [ 3938.858663] Lustre: Skipped 2 previous similar messages [ 3943.819508] Lustre: server umount lustre-MDT0000 complete [ 3947.930670] LustreError: 126551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842555 with bad export cookie 1466287636015775206 [ 3947.936347] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3947.943990] LustreError: 126551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3948.335313] Lustre: server umount lustre-MDT0001 complete [ 3957.122980] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 10:56:03 (1787842563) [ 3964.339774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.869905] LustreError: 130941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.892957] LustreError: 130941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3964.935082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.885269] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3970.026582] LustreError: 130942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.138703] LustreError: 130941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3979.661503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.179422] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 3993.463761] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4000.905767] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4009.465624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4010.018670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4010.033719] Lustre: Skipped 1 previous similar message [ 4018.477800] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4028.609871] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4033.733974] Lustre: *** cfs_fail_loc=159, val=0*** [ 4033.741845] Lustre: Skipped 1 previous similar message [ 4049.326457] Lustre: lustre-MDT0000: Client 2b83632e-6bd7-4b90-adf8-c0507727e81c (at 192.168.204.27@tcp) reconnecting [ 4051.055398] Lustre: *** cfs_fail_loc=159, val=0*** [ 4051.068708] Lustre: Skipped 1 previous similar message [ 4066.716292] Lustre: lustre-MDT0000: Client 2b83632e-6bd7-4b90-adf8-c0507727e81c (at 192.168.204.27@tcp) reconnecting [ 4075.494526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4075.496696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4075.519763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4075.538132] Lustre: Skipped 1 previous similar message [ 4079.793951] Lustre: server umount lustre-OST0000 complete [ 4085.736431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.741420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4085.756290] Lustre: Skipped 2 previous similar messages [ 4090.016606] Lustre: server umount lustre-MDT0000 complete [ 4090.862298] LustreError: 130942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.910379] LustreError: 130942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4094.342356] LustreError: 130922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842702 with bad export cookie 1466287636015776088 [ 4094.352543] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4094.363540] LustreError: 130922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4094.658289] Lustre: server umount lustre-MDT0001 complete [ 4105.974915] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4108.960391] Key type lgssc unregistered [ 4109.316073] LNet: 134718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4109.325727] LNetError: 134718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4109.342927] LNet: Removed LNI 192.168.204.127@tcp [ 4110.254091] Key type .llcrypt unregistered [ 4110.256747] Key type ._llcrypt unregistered [ 4134.831962] Key type ._llcrypt registered [ 4134.841375] Key type .llcrypt registered [ 4134.997771] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4136.221447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4136.412109] alg: No test for adler32 (adler32-zlib) [ 4137.540514] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 4137.793742] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4139.488563] Key type lgssc registered [ 4140.792888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4149.195952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4149.222051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.765282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.483198] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4160.993995] LustreError: 136128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.005541] LustreError: 136128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4165.824777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.190804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4170.621810] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4178.813370] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4184.442350] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4191.919324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4192.441907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4195.541988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4200.288466] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4210.331909] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4219.263867] Lustre: *** cfs_fail_loc=159, val=0*** [ 4235.660395] Lustre: lustre-MDT0000: Client 5fd07502-8dc2-4c16-a827-6651bed01e1a (at 192.168.204.27@tcp) reconnecting [ 4238.392850] Lustre: *** cfs_fail_loc=159, val=0*** [ 4254.599750] Lustre: lustre-MDT0001: Client 5fd07502-8dc2-4c16-a827-6651bed01e1a (at 192.168.204.27@tcp) reconnecting [ 4256.313247] Lustre: *** cfs_fail_loc=159, val=0*** [ 4256.324023] Lustre: Skipped 4 previous similar messages [ 4272.501879] Lustre: lustre-MDT0001: Client 5fd07502-8dc2-4c16-a827-6651bed01e1a (at 192.168.204.27@tcp) reconnecting [ 4278.223821] Lustre: 139179:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4281.535320] Lustre: *** cfs_fail_loc=159, val=0*** [ 4281.539394] Lustre: Skipped 1 previous similar message [ 4298.135407] Lustre: lustre-MDT0000: Client b8180ff4-f2ab-4bc5-bf1c-6f30fd03f210 (at 192.168.204.27@tcp) reconnecting [ 4300.186400] Lustre: *** cfs_fail_loc=159, val=0*** [ 4300.189971] Lustre: Skipped 14 previous similar messages [ 4316.538660] Lustre: lustre-MDT0000: Client b8180ff4-f2ab-4bc5-bf1c-6f30fd03f210 (at 192.168.204.27@tcp) reconnecting [ 4318.315160] Lustre: 139471:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4339.180333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4339.185492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.198744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4339.226645] Lustre: Skipped 1 previous similar message [ 4343.235915] Lustre: server umount lustre-OST0000 complete [ 4349.412853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.413891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4349.430575] Lustre: Skipped 1 previous similar message [ 4349.440969] Lustre: Skipped 2 previous similar messages [ 4353.721113] Lustre: server umount lustre-MDT0000 complete [ 4354.530510] LustreError: 137140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.564182] LustreError: 137140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4358.328018] LustreError: 136111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842966 with bad export cookie 3011264579058139490 [ 4358.332115] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4358.353725] LustreError: 136111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4358.827582] Lustre: server umount lustre-MDT0001 complete [ 4372.816787] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4376.440921] Key type lgssc unregistered [ 4376.717929] LNet: 140549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4376.747335] LNetError: 140549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4376.787057] LNet: Removed LNI 192.168.204.127@tcp [ 4378.270083] Key type .llcrypt unregistered [ 4378.274399] Key type ._llcrypt unregistered [ 4406.983566] Key type ._llcrypt registered [ 4406.987143] Key type .llcrypt registered [ 4407.101120] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4408.318982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4408.408803] alg: No test for adler32 (adler32-zlib) [ 4409.483219] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 4409.787779] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4411.512172] Key type lgssc registered [ 4412.780668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4422.039961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4422.077770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4423.546407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4428.438502] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4433.906312] LustreError: 141960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.933406] LustreError: 141960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4438.696724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.007620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4443.640590] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4451.900124] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4458.407290] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4465.034280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4465.313775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4468.414382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4471.598533] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4479.124810] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4483.776626] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4489.140517] Lustre: 144545:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4492.432226] Lustre: 144594:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4493.961543] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4495.127849] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4498.843424] Lustre: 144741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4499.716284] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4500.474458] Lustre: 144849:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4509.666525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4509.675530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4509.675948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4509.685435] Lustre: Skipped 1 previous similar message [ 4511.008979] Lustre: server umount lustre-OST0000 complete [ 4515.809603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4515.818575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.835146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4515.840074] Lustre: Skipped 1 previous similar message [ 4519.913319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.915675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4521.662615] Lustre: server umount lustre-MDT0000 complete [ 4525.026099] LustreError: 144263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.050751] LustreError: 144263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4525.447380] LustreError: 141941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843133 with bad export cookie 3851139745029501574 [ 4525.449655] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4525.461220] LustreError: 141941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4525.801436] Lustre: server umount lustre-MDT0001 complete [ 4539.932947] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4543.817460] Key type lgssc unregistered [ 4544.115469] LNet: 145904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4544.129406] LNetError: 145904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4545.192739] LNet: Removed LNI 192.168.204.127@tcp [ 4546.470306] Key type .llcrypt unregistered [ 4546.477893] Key type ._llcrypt unregistered [ 4579.015379] Key type ._llcrypt registered [ 4579.018102] Key type .llcrypt registered [ 4579.097272] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4580.645410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4580.666071] alg: No test for adler32 (adler32-zlib) [ 4581.761287] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 4582.114992] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4583.800287] Key type lgssc registered [ 4585.279891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4595.329620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4595.393131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.157657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.609922] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4606.443138] LustreError: 147317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.464346] LustreError: 147317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4611.569727] LustreError: 147318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.170902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.601100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4620.421001] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4629.392980] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4636.129538] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4644.423076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4644.976091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4651.686704] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4659.318519] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4660.369469] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4666.510301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4672.810146] Lustre: *** cfs_fail_loc=159, val=0*** [ 4672.814463] Lustre: Skipped 4 previous similar messages [ 4688.225966] Lustre: lustre-MDT0000: Client 0bbeb040-6790-44b6-bb20-141c73f27595 (at 192.168.204.27@tcp) reconnecting [ 4694.003532] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4694.009092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4694.019975] LustreError: Skipped 3 previous similar messages [ 4694.073101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4695.521465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4695.532045] LustreError: Skipped 4 previous similar messages [ 4695.536816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.549779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4696.040648] Lustre: server umount lustre-OST0000 complete [ 4701.673607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4701.693698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4701.705383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4707.654169] Lustre: server umount lustre-MDT0000 complete [ 4707.816302] LustreError: 148050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4707.834636] LustreError: 148050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4712.165721] LustreError: 147299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843320 with bad export cookie 6422617641144055384 [ 4712.172507] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4712.182124] LustreError: 147299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4712.717722] Lustre: server umount lustre-MDT0001 complete [ 4725.483350] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4728.811340] Key type lgssc unregistered [ 4729.057306] LNet: 151008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4729.066915] LNetError: 151008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4729.089211] LNet: Removed LNI 192.168.204.127@tcp [ 4730.038934] Key type .llcrypt unregistered [ 4730.040897] Key type ._llcrypt unregistered [ 4756.876674] Key type ._llcrypt registered [ 4756.879413] Key type .llcrypt registered [ 4757.040910] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 4758.039559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4758.216108] alg: No test for adler32 (adler32-zlib) [ 4759.465208] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 4759.745199] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 4761.432168] Key type lgssc registered [ 4762.894046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4771.905166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4771.919952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.387699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4779.677086] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4783.590348] LustreError: 152417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4783.614881] LustreError: 152417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4788.708817] LustreError: 152416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4791.569545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4792.523858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4797.854372] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4807.239495] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4813.874529] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4821.827175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4822.348663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4828.866707] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 4837.591599] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4837.762521] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4843.945578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4869.419637] Lustre: lustre-MDT0001: evicting a9eecfc2-ac40-4857-b633-4d900b4ae49b (at 192.168.204.27@tcp) by administrative request [ 4869.428218] Lustre: Skipped 1 previous similar message [ 4899.319702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4899.334559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4899.363797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4900.322056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4900.327284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4900.329768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4904.240913] Lustre: server umount lustre-OST0000 complete [ 4910.049198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4910.057179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.082243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4910.089398] Lustre: Skipped 1 previous similar message [ 4914.743138] Lustre: server umount lustre-MDT0000 complete [ 4915.684387] LustreError: 152417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.708797] LustreError: 152417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4919.581539] LustreError: 152397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843527 with bad export cookie 17751910276600013806 [ 4919.604345] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4919.980479] Lustre: server umount lustre-MDT0001 complete [ 4932.155044] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 4935.245330] Key type lgssc unregistered [ 4935.599694] LNet: 156508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4935.609403] LNetError: 156508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4936.678868] LNet: Removed LNI 192.168.204.127@tcp [ 4937.700385] Key type .llcrypt unregistered [ 4937.705194] Key type ._llcrypt unregistered [ 5029.261947] Key type ._llcrypt registered [ 5029.267853] Key type .llcrypt registered [ 5029.441813] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5030.796406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5030.870880] alg: No test for adler32 (adler32-zlib) [ 5032.176746] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 5032.452754] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5034.193860] Key type lgssc registered [ 5035.748853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5045.029403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5045.057128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.753718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.613452] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5056.995939] LustreError: 159017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5057.029805] LustreError: 159017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5062.124545] LustreError: 159016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.250228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.723377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5069.170389] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5077.988162] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5084.551551] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5093.638327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5094.155517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5101.694550] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5108.538124] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5111.005261] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5116.737059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5127.138355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5127.152350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.163079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5127.195783] Lustre: Skipped 1 previous similar message [ 5132.260033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.271051] Lustre: Skipped 1 previous similar message [ 5132.558210] Lustre: server umount lustre-OST0000 complete [ 5140.449703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5140.466718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5140.502081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5142.505059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5142.547304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5142.557383] Lustre: Skipped 1 previous similar message [ 5144.206335] Lustre: server umount lustre-MDT0000 complete [ 5147.627648] LustreError: 159016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5147.650489] LustreError: 159016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5148.471446] LustreError: 160870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843756 with bad export cookie 14104076609624593877 [ 5148.483763] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5148.485635] LustreError: 160870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5148.856938] Lustre: server umount lustre-MDT0001 complete [ 5162.921553] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5166.672585] Key type lgssc unregistered [ 5166.982624] LNet: 162578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5166.993939] LNetError: 162578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5167.020318] LNet: Removed LNI 192.168.204.127@tcp [ 5168.105773] Key type .llcrypt unregistered [ 5168.109280] Key type ._llcrypt unregistered [ 5199.623066] Key type ._llcrypt registered [ 5199.625388] Key type .llcrypt registered [ 5199.746205] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5200.822924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5201.028323] alg: No test for adler32 (adler32-zlib) [ 5202.309300] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 5202.526337] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5204.193168] Key type lgssc registered [ 5205.457921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5225.833988] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 11:17:11 (1787843831) [ 5245.562770] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 5257.813893] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5304.621468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5304.678992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.996181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5306.023107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5306.118496] Lustre: lustre-MDT0000: new disk, initializing [ 5306.255913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.282321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5312.173465] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5326.994704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5327.180376] Lustre: 166840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5327.284937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5327.303809] Lustre: Skipped 1 previous similar message [ 5327.415379] Lustre: lustre-MDT0001: new disk, initializing [ 5327.546891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5327.614325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5327.641112] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5333.575550] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5345.954186] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5353.633468] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5362.176869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5362.586673] Lustre: lustre-OST0000: new disk, initializing [ 5362.591683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5362.604634] Lustre: 168813:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5362.730468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5368.371344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5368.393531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5368.533168] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5370.269570] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5381.224873] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5394.405507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5394.422432] LustreError: Skipped 1 previous similar message [ 5394.423970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.441218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5394.447142] Lustre: Skipped 1 previous similar message [ 5399.373937] Lustre: server umount lustre-OST0000 complete [ 5404.601429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.605526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5404.636358] Lustre: Skipped 1 previous similar message [ 5404.639030] Lustre: Skipped 1 previous similar message [ 5409.763741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5409.766197] Lustre: Skipped 1 previous similar message [ 5411.151920] Lustre: server umount lustre-MDT0000 complete [ 5415.851492] LustreError: 166834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844023 with bad export cookie 15370425763245004368 [ 5415.854719] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5415.858629] LustreError: 166834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5416.296803] Lustre: server umount lustre-MDT0001 complete [ 5429.240871] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 5432.147175] Key type lgssc unregistered [ 5432.520525] LNet: 170717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5432.534352] LNetError: 170717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5432.553944] LNet: Removed LNI 192.168.204.127@tcp [ 5433.467645] Key type .llcrypt unregistered [ 5433.470296] Key type ._llcrypt unregistered [ 5460.639795] Key type ._llcrypt registered [ 5460.646291] Key type .llcrypt registered [ 5460.769667] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 5461.777922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5461.959502] alg: No test for adler32 (adler32-zlib) [ 5463.002195] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 5463.263044] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 5464.928642] Key type lgssc registered [ 5466.179516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5475.015899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5475.036609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.582729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.100920] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5487.079781] LustreError: 172144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5487.110200] LustreError: 172144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5492.211250] LustreError: 172145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.273517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5492.830232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5497.116285] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5505.833582] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5512.366408] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5519.398139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5519.807948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5521.867098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5526.567814] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 5534.942886] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5542.649117] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5542.667057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.683338] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5542.691305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.707331] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5543.830250] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5543.857223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5543.874289] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5543.884127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.934827] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5545.282086] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5545.305608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.314294] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5545.319625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.334297] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5546.371538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.553101] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5547.560878] Lustre: Skipped 1 previous similar message [ 5547.579037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.594580] Lustre: Skipped 1 previous similar message [ 5547.605165] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5547.612794] Lustre: Skipped 1 previous similar message [ 5547.618596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.627790] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5547.636837] Lustre: Skipped 1 previous similar message [ 5548.796385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.874505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.668046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.634889] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5551.639901] Lustre: Skipped 3 previous similar messages [ 5551.651775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.660908] Lustre: Skipped 3 previous similar messages [ 5551.664691] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5551.669939] Lustre: Skipped 3 previous similar messages [ 5551.673338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.682118] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5551.689188] Lustre: Skipped 3 previous similar messages [ 5552.542975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.519867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.398688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.296967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.157627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.116560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.979994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.775732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.572862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.405123] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5560.417153] Lustre: Skipped 9 previous similar messages [ 5560.429567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5560.438946] Lustre: Skipped 9 previous similar messages [ 5560.442290] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5560.448411] Lustre: Skipped 9 previous similar messages [ 5560.451742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.458843] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5560.468824] Lustre: Skipped 9 previous similar messages [ 5561.354923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.320911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.388558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.641342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.492477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.372708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.121283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.028511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.997382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.729234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.558521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.533392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.412603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.352307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.268306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.100836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.947615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.829775] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5576.837962] Lustre: Skipped 17 previous similar messages [ 5576.854712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5576.863639] Lustre: Skipped 17 previous similar messages [ 5576.868231] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5576.872590] Lustre: Skipped 17 previous similar messages [ 5576.876431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.885832] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5576.892890] Lustre: Skipped 17 previous similar messages [ 5577.806835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.714346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.644821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.660208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.441671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.352990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.115412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.136364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.988431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.960979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.794831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.672701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.534267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.395586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.247381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.096303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.979064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.010400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.828479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.913901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.845904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.780269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.683747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.689601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.474633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.451611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.293255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.138937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.419365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.435468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.407608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.355483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.322413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.287592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.008274] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5609.013361] Lustre: Skipped 34 previous similar messages [ 5609.023626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.033723] Lustre: Skipped 34 previous similar messages [ 5609.037872] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5609.041832] Lustre: Skipped 34 previous similar messages [ 5609.045269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.055038] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5609.060327] Lustre: Skipped 34 previous similar messages [ 5609.828952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.698337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.510218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.397369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.248420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.057516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.984429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.918161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.834577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.691896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.639837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.584507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.459591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.467564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.401715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.352681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.302766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.361409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.417679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.366626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.292530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.332606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.281927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.279672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.229088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.307228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.080735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.340286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.345342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.172967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.016888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.018788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.855996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.826077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.694252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.737022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.694831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.782321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.762631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.627267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.458214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.377991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.159289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.050785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.041578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.955979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.790777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.602950] LustreError: 172164:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5653.614765] LustreError: 172164:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5653.618658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.591271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.505243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.497110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.484915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.405868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.142325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.937454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.726660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.588375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.385808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.307520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.138176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.936086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.753970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.536595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.308827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.183713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.067305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.792638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.585072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.449178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.278391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.114286] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5673.120548] Lustre: Skipped 70 previous similar messages [ 5673.132433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.139419] Lustre: Skipped 70 previous similar messages [ 5673.142559] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5673.146848] Lustre: Skipped 70 previous similar messages [ 5673.149812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.160194] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5673.164486] Lustre: Skipped 70 previous similar messages [ 5674.147476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.958807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.012825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.977279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.881197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.728446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.703792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.542848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.365528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.361781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.401202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.425572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.222835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.214961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.342495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.239624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.391574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.318579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.349241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.245359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.204502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.148207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.159508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.136138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.401154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.484942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.517669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.469707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.502779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.612141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.686858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.686516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.796803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.655953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.797857] LustreError: 172164:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5707.809310] LustreError: 172164:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5707.815472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.695287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.504865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.592106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.534826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.412217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.682217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.661920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.656872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.509579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.442922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.338787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.284582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.385148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.458409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.356346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.124986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.122707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.176476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.144165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.270747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.391411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.574722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.724637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.830925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.898597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.973210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.019361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.000456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.089320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.898852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.847266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.721767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.728109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.685414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.702590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.676908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.976828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.076767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.222830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.559870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.822057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.988630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.096083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.204690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.272371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.355414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.510912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.582135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.811098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.973447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.088926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.116495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.048456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.140792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.524496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.876653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.093599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.327560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.236886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.263066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.172434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.301363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.461790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.662208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.984368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.210024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.328503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.569896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.788515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.067686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.039572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.154777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.207315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.077537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.935366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.056877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.054187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.054493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.181740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.372433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.608850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.113463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.368529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.619093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.973521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.181993] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5801.188972] Lustre: Skipped 120 previous similar messages [ 5801.211628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5801.220671] Lustre: Skipped 120 previous similar messages [ 5801.224662] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5801.229746] Lustre: Skipped 120 previous similar messages [ 5801.233366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.244322] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5801.253332] Lustre: Skipped 120 previous similar messages [ 5802.738458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.089804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.269157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.570971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.908804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.275122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.616915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.124849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.582971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.033596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.105434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.182448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.153262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.246374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.220406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.356869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.404202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.881610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.280583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.561708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.851355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.785051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.976745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.152722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.230172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.440968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.414110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.513329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.704563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.025436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.089857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.347759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.762179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.792959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.757336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.695204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.744817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.694080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.642246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.470114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.355804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.396901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.335345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.476522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.442959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.675903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.737068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.807808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.961420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.824882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.995637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.941912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.956929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.055103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.957857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.099902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.139122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.327341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.432741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.463664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.448150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.429336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.470141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.508991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.579030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.529678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.479533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.450924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.325138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.279944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.168501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.319535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.179300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.129267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.103516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.059189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.099937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.027764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.948959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.001221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.909409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.149969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.269431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.269092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.252433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.269109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.391867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.553758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.644073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.688039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.643377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.737709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.707235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.732096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.599452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.468303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.570224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.619192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.404767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.372689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.722276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.971861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.989193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.179995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.306403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.453613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.712058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.956312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.306711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.474845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.844754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.023264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.042465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.102238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.280940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.512433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.881768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.384095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.965737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.145269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.310643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.608720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.729636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.035996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.940401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.993934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.894385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.054699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.233594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.763072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.634147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.258330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.856778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.684725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.681061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.023082] LustreError: 172164:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5953.039767] LustreError: 172164:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5953.046132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.957995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.734181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.784496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.267550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.429155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.139626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.133745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.895364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.899069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.745966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.686129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.710748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.699361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.766404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.724747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.840829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.932064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.735397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.748243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.799151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.758344] LustreError: 172164:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5973.774628] LustreError: 172164:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5973.777291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.656799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.646033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.632981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.528928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.356064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.199426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.984456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.954986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.114273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.240339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.345099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.455965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.272393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.194591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.026501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.082804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.120931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.153405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.976435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.016535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.999456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.947809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.811576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.720115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.714265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.635197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.618043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.667152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.619742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.687123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.659460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.548343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.518940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.531191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.403450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.348295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.544496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.627096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.659296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.872523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.084903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.967488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.046847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.963869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.896915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.134279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.216642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.568441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.488536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.549665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.528107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.631771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.456465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.492347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.511429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.408180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.317553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.390288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.509944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.527827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.758776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.711179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.024891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.239781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.637728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.658794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.692807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.725092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.917424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.079196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.084697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.589995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.637900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.971499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.175668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.152897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.185688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.342271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.529781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.662963] LustreError: 172164:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6056.677957] LustreError: 172164:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6056.678230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.679593] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6057.685091] Lustre: Skipped 237 previous similar messages [ 6057.699989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6057.712830] Lustre: Skipped 237 previous similar messages [ 6057.717201] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6057.721173] Lustre: Skipped 237 previous similar messages [ 6057.728684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.747130] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6057.756710] Lustre: Skipped 237 previous similar messages [ 6058.785687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.140761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.233870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.339531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.478777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.514664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.857249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.125361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.243360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.350692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.419716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.412690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.398501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.599541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.762711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.970752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.343444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.448608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.324840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.459317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.480667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.537293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.571157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.614449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.631811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.518675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.542677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.519087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.556033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.460172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.373584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.406579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.680405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.741953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.117848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.228654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.200054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.331780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.351470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.678982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.799510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.984679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.234979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.400650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.959530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.092891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.169815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.468979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.344135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.488751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.415922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.566984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.712308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.883497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.805631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.101462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.081469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.369041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.492707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.448678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.617916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.952592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.225051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.490596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.706884] LustreError: 172164:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6130.721603] LustreError: 172164:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6130.749510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.912797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.914560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.151039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.309817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.351152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.284139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.054317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.067728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.857368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.873522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.925937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.705332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.621026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.667736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.909335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.949578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.860578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.812677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.934775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.150193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.331151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.398858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.321445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.353879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.378450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.301463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.214098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.426777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.375817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.300910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.232544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.167285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.098830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.126907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.012328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.146046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.227688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.531103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.794598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.751309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.964558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.088556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.156403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.336685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.607805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.755744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.938584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.053603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.264075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.237709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.491746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.499919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.662078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.750589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.950944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.299683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.462450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.529365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.713273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.881957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.938077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.216919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.049670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.205755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.198320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.242197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.394197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.455797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.617066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.085114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.299538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.721107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.030764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.222801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.239616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.159432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.109060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.975655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.961354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.019674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.940413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.855982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.763131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.941532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.951593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.007667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.267876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.474025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.563953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.636570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.663786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.751975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.685932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.751063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.770036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.897948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.067393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.120646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.102963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.010238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.061314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.045732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.023020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.918753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.035971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.040283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.443444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.573728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.677595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.111632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.440366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.459189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.670647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.664695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.833972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.018100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.274919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.258666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.399215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.351475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.202561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.073933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.956795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.119199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.151161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.129094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.432973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.589899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.706959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.751781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.829065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.955970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.896320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.005784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.989615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.061417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.105202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.256870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.432322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.550103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.715246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.873616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.433581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.845886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.050680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.219991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.481675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.957167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.368764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.539161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.961874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.197925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.765816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.543273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.046720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.072192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.714558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.293353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.672885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.954814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.453443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.916343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.296771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.708880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.977165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.079885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.797640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.859902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.698695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.995486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.027575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.978193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.206776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.167069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.163782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.654305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.723824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.699623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.833229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.409815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.551430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.569403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.586040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.444718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.451385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.428347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.433804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.296531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.394374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.231967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.036908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.842787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.927559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.265891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.557439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.018338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.906130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.146333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.383003] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.437728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.674392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.938812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.938319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.924609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.152494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.347540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.225543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.398304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.632171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.501167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.669734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.518933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.358926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.240846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.363565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.346811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.357185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.221658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.441072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.514463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.525550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.593540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.566911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.544711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.686685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.676644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.938846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.096544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.039982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.949041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.775116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.684602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.479822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.434399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.232697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.170757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.011761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.017709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.988844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.042702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.846670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.661220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.723662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.713271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.957693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.803678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.761473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.746132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.690320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.578399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.670205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.631140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.570959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.479559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.555076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.559757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.714985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.906938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.019697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.104361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.180250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.009248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.999298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.016488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.889969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.734939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.517950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.359843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.179921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.060603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.895040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.007905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.984398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.078573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.011621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.044520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.100790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.315604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.201117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.031436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.931641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.934273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.138316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.393475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.520181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.544556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.747899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.844102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.072467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.000839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.081140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.192026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.130188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.568187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.444475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.720410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.801315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.042987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.170188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.192772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.120347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.029170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.921121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.127049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.333347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.628984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.732882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.804856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.666089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.692502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.645682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.638827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.482239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.677653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.608490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.506311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.441288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.724669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.586696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.484737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.687502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.751929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.672726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.962218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.056644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.371508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.537251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.759555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.431063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.619978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.955464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.035439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.240608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.538264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.563314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.629794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.890693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.494455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.750476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.126940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.895810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.784621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.536408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.040278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.090586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.239295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.582422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.661700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.505418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.334089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.359647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.476081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.667899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.708555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.884573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.945930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.073067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.084114] LustreError: 172164:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6519.091772] LustreError: 172164:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6519.121832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.063413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.049817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.019367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.099939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.281776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.600702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.748396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.724619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.788880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.721610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.583905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.445705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.482284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.420154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.367626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.193253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.054884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.032733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.935752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.887077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.986596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.992856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.977230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6543.017420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.093401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.926712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.980823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.879877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.728559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.648427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.446789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.253162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.247491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6552.268292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.031485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.770558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.567945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.540664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.534733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.644712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.520481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.378624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.469633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.396695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.441338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.271928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.523366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6565.504780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.490830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.394316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.247588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.443739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6570.675095] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6570.679239] Lustre: Skipped 475 previous similar messages [ 6570.695599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.707274] Lustre: Skipped 475 previous similar messages [ 6570.710886] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6570.717469] Lustre: Skipped 475 previous similar messages [ 6570.722106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6570.735351] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6570.740081] Lustre: Skipped 475 previous similar messages [ 6571.769163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6572.900471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6573.944993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.993274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.029658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.970571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6578.079191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6579.091396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.205628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.159902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.088848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.992200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.890979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6584.830510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6585.853032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6586.753991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.928767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6588.987765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.918302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6591.218680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.333551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.456916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7043.740987] LustreError: 173968:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 7043.767574] LustreError: 172164:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7814.626829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7814.643034] Lustre: Skipped 22 previous similar messages [ 7814.650113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7814.657617] Lustre: Skipped 1 previous similar message [ 7819.756520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7819.778505] Lustre: Skipped 1 previous similar message [ 7821.238623] Lustre: server umount lustre-OST0000 complete [ 7829.987378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7830.004322] Lustre: Skipped 1 previous similar message [ 7835.107378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7835.125178] Lustre: Skipped 1 previous similar message [ 7840.229803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7840.237124] Lustre: Skipped 1 previous similar message [ 7842.272714] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7842.686500] Lustre: server umount lustre-MDT0000 complete [ 7844.336219] LustreError: 174581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7844.361630] LustreError: 174581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7847.036926] LustreError: 172878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846454 with bad export cookie 13734674132643609373 [ 7847.052897] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7847.069598] LustreError: 172878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7847.814806] Lustre: server umount lustre-MDT0001 complete [ 7861.337621] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 7864.341758] Key type lgssc unregistered [ 7864.671673] LNet: 227455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7864.678305] LNetError: 227455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7864.702577] LNet: Removed LNI 192.168.204.127@tcp [ 7865.625320] Key type .llcrypt unregistered [ 7865.627626] Key type ._llcrypt unregistered [ 7893.278917] Key type ._llcrypt registered [ 7893.284243] Key type .llcrypt registered [ 7893.420555] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 7894.675388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7894.805000] alg: No test for adler32 (adler32-zlib) [ 7895.861087] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 7896.123393] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 7897.813143] Key type lgssc registered [ 7899.117039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7907.203146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7907.220625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.620539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7913.531245] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7919.075939] LustreError: 228871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.090854] LustreError: 228871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7923.514918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7923.849391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7929.429222] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7938.814071] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7946.215897] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7954.660355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7955.165805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7962.312837] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 7971.314970] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7978.737697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7987.545792] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7989.746587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.766365] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7989.771624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7989.784808] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7989.788432] LustreError: 228890:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7993.720346] Lustre: lustre-OST0000: Not available for connect from 192.168.204.27@tcp (stopping) [ 7994.337331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7994.343186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.367238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7994.375667] Lustre: Skipped 1 previous similar message [ 7994.404846] Lustre: Skipped 1 previous similar message [ 7998.441787] Lustre: server umount lustre-OST0000 complete [ 8009.848193] LustreError: 228890:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 8033.851547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8034.250298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8035.333603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8035.352915] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8035.356157] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8035.389080] Lustre: Skipped 1 previous similar message [ 8042.164577] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8051.117238] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8060.738990] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 8067.299962] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99568536d000.ost_server_uuid 50 [ 8069.867076] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff99568536d000.ost_server_uuid in FULL state after 0 sec [ 8076.259201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8076.268344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8076.275827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8076.300071] Lustre: Skipped 1 previous similar message [ 8081.385899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8081.402931] Lustre: Skipped 2 previous similar messages [ 8082.460890] Lustre: server umount lustre-OST0000 complete [ 8091.621598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8091.623653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8091.645124] Lustre: Skipped 1 previous similar message [ 8101.858677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8101.875887] Lustre: Skipped 4 previous similar messages [ 8102.369814] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8102.974077] Lustre: server umount lustre-MDT0000 complete [ 8106.985059] LustreError: 229843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8107.011494] LustreError: 229843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8108.112958] LustreError: 228850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846715 with bad export cookie 10077446816834646119 [ 8108.117492] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8108.121428] LustreError: 228850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8108.620651] Lustre: server umount lustre-MDT0001 complete [ 8133.630915] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 8137.542135] Key type lgssc unregistered [ 8137.820195] LNet: 234127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8137.829331] LNetError: 234127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8137.846147] LNet: Removed LNI 192.168.204.127@tcp [ 8139.265250] Key type .llcrypt unregistered [ 8139.274647] Key type ._llcrypt unregistered [ 8184.653471] Key type ._llcrypt registered [ 8184.655121] Key type .llcrypt registered [ 8184.777177] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8185.849689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8185.859929] alg: No test for adler32 (adler32-zlib) [ 8187.354956] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 8187.669159] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 8189.435220] Key type lgssc registered [ 8191.002120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8203.159939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8203.195584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8204.856990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8210.534292] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8215.011396] LustreError: 235867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8215.048939] LustreError: 235867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8220.136895] LustreError: 235868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8220.247292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8220.743129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8225.553556] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8228.729122] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8235.647717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8235.974351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8242.020413] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8249.276797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8251.187581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8251.393150] Lustre: lustre-OST0001: new disk, initializing [ 8251.408113] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8251.423301] Lustre: 238175:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8251.522270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8257.981422] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8260.795510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8260.842708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8261.057830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8270.295847] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8278.919402] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8296.967454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.981732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.003549] Lustre: Skipped 3 previous similar messages [ 8297.030833] Lustre: Skipped 3 previous similar messages [ 8302.079868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8302.101406] Lustre: Skipped 1 previous similar message [ 8304.194591] Lustre: server umount lustre-MDT0000 complete [ 8307.182249] LustreError: 235867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8310.593262] LustreError: 235846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846918 with bad export cookie 6725684481990578379 [ 8310.599172] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8310.605295] LustreError: 235846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8311.139963] Lustre: server umount lustre-MDT0001 complete [ 8326.916125] Lustre: server umount lustre-OST0000 complete [ 8328.544277] Lustre: 235128:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846920/real 1787846920] req@ffff9e0f42f40e00 x1874693241013120/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8328.584648] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8331.201187] Lustre: server umount lustre-OST0001 complete [ 8355.125300] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8367.425669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8368.078455] LustreError: 241282:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8368.115685] LustreError: 241282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8368.202236] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8373.218707] LustreError: 241283:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.616748] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8382.436166] LustreError: 241282:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8382.451047] LustreError: 241282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8386.410803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.123357] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8392.537331] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8396.294468] Lustre: 242425:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8404.382477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.906385] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8411.724686] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8412.077292] LustreError: 242778:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8412.087385] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 8414.140462] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8421.582540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8427.016438] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 8428.916339] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8438.174586] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8442.692869] Lustre: 244296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8457.705162] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.705975] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8457.724236] Lustre: Skipped 2 previous similar messages [ 8457.739823] Lustre: Skipped 5 previous similar messages [ 8460.859695] Lustre: server umount mylustre-MDT0000 complete [ 8462.822754] LustreError: 241279:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8462.851932] LustreError: 241279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 8465.191439] LustreError: 244496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847073 with bad export cookie 6725684481990581242 [ 8465.192127] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8465.209500] LustreError: 244496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8465.845701] Lustre: server umount mylustre-MDT0001 complete [ 8481.308820] Lustre: server umount mylustre-OST0000 complete [ 8483.296094] Lustre: 235128:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787847075/real 1787847075] req@ffff9e0f440dfb80 x1874693241081216/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787847091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8483.353958] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8483.370569] Lustre: Skipped 1 previous similar message [ 8485.886544] Lustre: server umount mylustre-OST0001 complete [ 8510.587754] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8523.016597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.626494] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8523.637358] Lustre: Skipped 1 previous similar message [ 8528.869921] LustreError: 246607:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8528.882267] LustreError: 246607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8529.035780] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8542.436732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8548.540549] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8552.935305] Lustre: 247750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8564.604845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8567.404967] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 8568.438770] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8573.772459] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8584.404512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8589.820682] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8590.691804] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8598.169582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8601.513671] Lustre: 249621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8609.764946] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8609.772246] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8609.790573] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8615.138437] Lustre: server umount tfs-MDT0000 complete [ 8618.825601] LustreError: 247353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847226 with bad export cookie 6725684481990583419 [ 8618.830124] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8618.838284] LustreError: 247353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8619.137558] Lustre: server umount tfs-MDT0001 complete [ 8632.248644] Lustre: server umount tfs-OST0000 complete [ 8646.938291] Lustre: server umount tfs-OST0001 complete [ 8674.192889] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8685.411420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8686.134371] LustreError: 251873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8686.167818] LustreError: 251873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8686.266300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8686.281210] Lustre: Skipped 3 previous similar messages [ 8691.503193] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8701.395338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8707.145211] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8710.537233] Lustre: 253014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8718.078590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8725.081578] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8727.733601] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 8732.897921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8734.672509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8740.358288] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8741.308318] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8749.957594] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8753.903316] Lustre: 254886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8762.233362] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 12:16:08 (1787847368) [ 8781.290062] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8781.299076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8781.317785] Lustre: Skipped 5 previous similar messages [ 8781.327944] Lustre: Skipped 7 previous similar messages [ 8785.478769] Lustre: server umount lustre-MDT0000 complete [ 8789.403336] LustreError: 251848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847397 with bad export cookie 6725684481990585337 [ 8789.421778] LustreError: 251848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8789.424761] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8791.531698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8795.849048] Lustre: server umount lustre-MDT0001 complete [ 8806.351590] Lustre: server umount lustre-OST0000 complete [ 8825.314717] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8825.645367] Lustre: server umount lustre-OST0001 complete [ 8833.865348] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 8848.733601] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 8891.129304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8891.382773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8891.408493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8891.488951] Lustre: lustre-MDT0000: new disk, initializing [ 8891.580544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8891.585297] Lustre: Skipped 3 previous similar messages [ 8891.605215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8896.373270] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8909.110404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8909.246225] Lustre: 258094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8909.293334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8909.295922] Lustre: Skipped 1 previous similar message [ 8909.353981] Lustre: lustre-MDT0001: new disk, initializing [ 8909.477324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8909.494348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8914.475526] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8926.126993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8926.531208] Lustre: lustre-OST0000: new disk, initializing [ 8926.535445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8926.549311] Lustre: 259675:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8927.922366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8927.944924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8928.170391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8934.663425] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8946.057179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8946.225178] Lustre: lustre-OST0001: new disk, initializing [ 8946.230876] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8946.240958] Lustre: 260546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8947.727663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8947.745510] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8947.885887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8953.468745] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 8966.731670] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 12:19:32 (1787847572) [ 8983.474546] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 12:19:49 (1787847589) [ 8999.407706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.428967] Lustre: Skipped 3 previous similar messages [ 8999.440085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8999.450289] Lustre: Skipped 4 previous similar messages [ 9004.719192] Lustre: server umount lustre-OST0000 complete [ 9009.638761] LustreError: 259659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9009.657260] LustreError: 259659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 9014.856146] Lustre: server umount lustre-MDT0000 complete [ 9018.817661] LustreError: 261622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847626 with bad export cookie 6725684481990587115 [ 9018.818642] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9018.829030] LustreError: 261622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9033.184179] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9033.491600] Lustre: server umount lustre-MDT0001 complete [ 9043.860402] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9120.418757] Lustre: server umount lustre-OST0001 complete [ 9127.729130] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 9143.482146] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9186.559864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9186.837239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9186.875671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9186.964519] Lustre: lustre-MDT0000: new disk, initializing [ 9187.055936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9187.067901] Lustre: Skipped 3 previous similar messages [ 9187.089350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9191.936327] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9205.681110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9205.866931] Lustre: 265404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9205.979800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9205.991041] Lustre: Skipped 1 previous similar message [ 9206.162674] Lustre: lustre-MDT0001: new disk, initializing [ 9206.421171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9206.437674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9211.652696] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9222.310755] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9228.631852] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9237.349549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9237.763859] Lustre: lustre-OST0000: new disk, initializing [ 9237.771742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9237.783205] Lustre: 267371:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9239.649737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9239.668816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9239.791833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9244.564841] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9255.027959] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9265.639903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9265.641147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9265.650877] LustreError: Skipped 1 previous similar message [ 9265.653899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9265.662156] Lustre: Skipped 6 previous similar messages [ 9265.667336] Lustre: Skipped 8 previous similar messages [ 9269.057681] Lustre: server umount lustre-OST0000 complete [ 9280.365536] Lustre: server umount lustre-MDT0000 complete [ 9284.399199] LustreError: 265394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847892 with bad export cookie 6725684481990588585 [ 9284.415195] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9284.425507] LustreError: 265394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9284.747543] Lustre: server umount lustre-MDT0001 complete [ 9296.979987] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9299.564671] Key type lgssc unregistered [ 9299.941987] LNet: 269131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9299.952824] LNetError: 269131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9299.967749] LNet: Removed LNI 192.168.204.127@tcp [ 9301.076350] Key type .llcrypt unregistered [ 9301.079333] Key type ._llcrypt unregistered [ 9328.975506] Key type ._llcrypt registered [ 9328.977946] Key type .llcrypt registered [ 9329.127259] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9330.003689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9330.183039] alg: No test for adler32 (adler32-zlib) [ 9331.300360] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 9331.539311] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9333.192258] Key type lgssc registered [ 9334.494258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9342.604481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9342.637295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9344.120455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9348.564905] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9353.699819] LustreError: 270592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.730669] LustreError: 270592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9356.988686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.591849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9362.638307] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9371.229398] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9380.017674] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9388.501578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9388.929405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9396.049743] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9404.301432] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9405.446920] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9412.581244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9412.585490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9412.616211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9413.673506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9413.855076] Lustre: server umount lustre-OST0000 complete [ 9418.723900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9418.749970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9418.762549] Lustre: Skipped 1 previous similar message [ 9418.775781] Lustre: Skipped 2 previous similar messages [ 9423.843635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9423.861891] Lustre: Skipped 1 previous similar message [ 9424.796662] Lustre: server umount lustre-MDT0000 complete [ 9428.301754] LustreError: 272435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848036 with bad export cookie 17508677555325767184 [ 9428.306397] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9428.319161] LustreError: 272435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9428.788467] Lustre: server umount lustre-MDT0001 complete [ 9440.886475] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9443.376233] Key type lgssc unregistered [ 9443.640789] LNet: 274174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9443.659115] LNetError: 274174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9443.684408] LNet: Removed LNI 192.168.204.127@tcp [ 9444.512749] Key type .llcrypt unregistered [ 9444.515472] Key type ._llcrypt unregistered [ 9462.954453] Key type ._llcrypt registered [ 9462.956105] Key type .llcrypt registered [ 9463.061488] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9464.098403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9464.159680] alg: No test for adler32 (adler32-zlib) [ 9465.214917] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 9465.471744] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9467.112321] Key type lgssc registered [ 9468.152622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9477.327240] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9477.349165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9492.897247] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9498.016294] LustreError: 275365:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.127@tcp: failed processing log, type 4: rc = -110 [ 9524.642499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9532.383489] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9541.598573] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9543.862477] Lustre: server umount lustre-OST0000 complete [ 9562.903374] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [ 9565.868609] Key type lgssc unregistered [ 9566.251602] LNet: 276997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9566.268791] LNetError: 276997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9566.285874] LNet: Removed LNI 192.168.204.127@tcp [ 9567.335306] Key type .llcrypt unregistered [ 9567.339723] Key type ._llcrypt unregistered [ 9611.813920] Key type ._llcrypt registered [ 9611.822843] Key type .llcrypt registered [ 9611.941932] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9613.043837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9613.161743] alg: No test for adler32 (adler32-zlib) [ 9614.140812] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [ 9614.331551] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [ 9616.008321] Key type lgssc registered [ 9617.195635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9632.634109] loop0: detected capacity change from 0 to 204800000 [ 9632.777125] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9632.784752] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9633.294111] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9634.600496] loop0: detected capacity change from 0 to 204800000 [ 9634.849461] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9634.874444] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9635.516984] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9636.811990] loop0: detected capacity change from 0 to 204800000 [ 9637.148463] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9637.160333] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9637.349564] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9638.466185] loop0: detected capacity change from 0 to 204800000 [ 9638.652187] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9638.662793] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9638.816985] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9639.965345] loop0: detected capacity change from 0 to 204800000 [ 9640.019818] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9644.865792] loop0: detected capacity change from 0 to 204800000 [ 9644.899483] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9649.505583] loop0: detected capacity change from 0 to 204800000 [ 9649.543611] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9654.881927] loop0: detected capacity change from 0 to 204800000 [ 9654.915577] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9659.064937] loop0: detected capacity change from 0 to 204800000 [ 9659.152549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9659.199963] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9659.286744] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9659.298188] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9659.307937] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9659.316216] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9659.327946] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9659.336628] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9659.350678] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9659.359802] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9659.371497] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9659.382308] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9659.392130] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9659.401503] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9659.415436] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9659.429927] Lustre: 280413:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9665.717772] Lustre: server umount MGS complete [ 9666.770172] loop0: detected capacity change from 0 to 204800000 [ 9666.946358] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9667.489555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9667.548903] Lustre: lustre-MDT0000: Aborting client recovery [ 9668.493678] loop1: detected capacity change from 0 to 204800000 [ 9668.674248] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.794644] Lustre: 280845:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9668.808522] Lustre: 280845:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9668.822671] Lustre: 280845:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9668.835922] Lustre: 280845:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9668.850293] Lustre: 280845:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9668.865160] Lustre: 280845:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9668.875739] Lustre: 280845:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9668.904228] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9669.075881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9669.133697] Lustre: lustre-MDT0001: Aborting client recovery [ 9670.076985] loop2: detected capacity change from 0 to 204800000 [ 9670.301948] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9670.363268] Lustre: 280925:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9670.379723] Lustre: 280925:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9670.413712] Lustre: 280925:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9670.442311] Lustre: 280925:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9670.459294] Lustre: 280925:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9670.478905] Lustre: 280925:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9670.497790] Lustre: 280925:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9670.697780] Lustre: Found index 0 for lustre-OST0000, updating log [ 9670.762774] Lustre: 280945:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9670.845373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9670.871953] Lustre: lustre-OST0000: Aborting recovery [ 9671.803984] loop3: detected capacity change from 0 to 204800000 [ 9671.946530] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9672.076747] Lustre: 281010:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9672.089468] Lustre: 281010:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9672.104871] Lustre: 281010:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9672.126953] Lustre: 281010:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9672.136690] Lustre: 281010:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9672.149735] Lustre: 281010:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9672.166851] Lustre: 281010:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9672.189898] Lustre: Found index 1 for lustre-OST0001, updating log [ 9672.235567] Lustre: 281015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9672.343519] Lustre: lustre-OST0001: Aborting recovery [ 9677.421232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9677.437154] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9677.468712] LustreError: 280789:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9677.487242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9677.526715] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9678.449466] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9678.463679] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9681.927658] LustreError: 280756:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 441us: rc = -2 [ 9682.437749] LustreError: 280754:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1671us: rc = -2 [ 9682.470204] LustreError: 280754:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 20 previous similar messages [ 9699.297320] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9699.301705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9699.312878] Lustre: Skipped 2 previous similar messages [ 9699.324256] Lustre: Skipped 1 previous similar message [ 9702.053562] Lustre: server umount lustre-MDT0000 complete [ 9703.032957] LustreError: 280741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848310 with bad export cookie 13368308581413572321 [ 9703.040026] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9703.044441] LustreError: 280741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9704.423064] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.426311] LustreError: 281247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9704.439182] Lustre: Skipped 2 previous similar messages [ 9704.445017] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9704.456839] LustreError: 281247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9704.459326] Lustre: Skipped 2 previous similar messages [ 9708.451453] LustreError: 280755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9708.452574] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9708.488662] Lustre: Skipped 2 previous similar messages [ 9709.569958] Lustre: server umount lustre-MDT0001 complete [ 9716.681322] Lustre: server umount lustre-OST0000 complete [ 9723.945914] Lustre: server umount lustre-OST0001 complete [ 9733.238876] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 12:32:19 (1787848339) [ 9762.152484] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [ 9774.937921] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [ 9817.051602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9817.202396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9817.225942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9817.286453] Lustre: lustre-MDT0000: new disk, initializing [ 9817.382386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9817.389515] Lustre: Skipped 1 previous similar message [ 9817.411817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9821.159235] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9832.598983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9832.767422] Lustre: 285082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9832.864478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9832.872618] Lustre: Skipped 1 previous similar message [ 9833.015624] Lustre: lustre-MDT0001: new disk, initializing [ 9833.133447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9833.221389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9833.232749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9838.838970] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9849.296396] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9855.387912] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9862.345156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9862.595871] Lustre: lustre-OST0000: new disk, initializing [ 9862.599670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9862.615215] Lustre: 287051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9862.710603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.959514] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [ 9870.397558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9870.421470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9870.529634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9879.194697] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9884.908361] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9898.483764] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9907.063308] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9915.084956] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9968.097083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9968.103991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9968.121528] Lustre: Skipped 1 previous similar message [ 9968.131114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9968.141098] Lustre: Skipped 1 previous similar message [ 9973.227549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9973.232943] Lustre: Skipped 1 previous similar message [ 9981.408117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9981.600640] Lustre: server umount lustre-OST0000 complete [ 9987.045401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9987.055392] LustreError: Skipped 1 previous similar message [ 9987.059875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9987.070163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9987.077632] Lustre: Skipped 2 previous similar messages [ 9992.549941] Lustre: server umount lustre-MDT0000 complete [ 9993.699131] LustreError: 287770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9993.718240] LustreError: 287770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9996.655343] LustreError: 285076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848604 with bad export cookie 13368308581413575037 [ 9996.657899] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9996.675562] LustreError: 285076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9997.168454] Lustre: server umount lustre-MDT0001 complete [10005.810347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10011.451704] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10017.624465] Lustre: server umount MGS complete [10030.575415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10030.953960] LustreError: 291022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10030.982339] LustreError: 291035:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10031.039549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10036.122405] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10036.194725] LustreError: 291023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10036.234159] LustreError: 291023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10045.410957] LustreError: 291022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10045.438312] LustreError: 291022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10046.535872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.209989] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10062.731961] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10069.657798] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10077.677845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10078.127863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10078.136708] Lustre: Skipped 1 previous similar message [10085.027638] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10093.297325] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10137.571339] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10137.581227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10137.596472] Lustre: Skipped 1 previous similar message [10137.602170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10137.604686] Lustre: Skipped 2 previous similar messages [10141.675683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10142.884282] Lustre: server umount lustre-OST0000 complete [10146.785643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10146.802528] Lustre: Skipped 2 previous similar messages [10152.136383] Lustre: server umount lustre-MDT0000 complete [10155.457958] LustreError: 291002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848763 with bad export cookie 13368308581413576990 [10155.460143] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10155.467527] LustreError: 291002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10155.873374] Lustre: server umount lustre-MDT0001 complete [10167.004839] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [10170.760728] Key type lgssc unregistered [10171.050658] LNet: 294988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10171.066600] LNetError: 294988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10171.100061] LNet: Removed LNI 192.168.204.127@tcp [10172.528760] Key type .llcrypt unregistered [10172.531629] Key type ._llcrypt unregistered [10218.097172] Key type ._llcrypt registered [10218.101841] Key type .llcrypt registered [10218.212953] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [10234.319501] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10235.357318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10235.496328] alg: No test for adler32 (adler32-zlib) [10236.514386] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [10236.749827] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [10238.408191] Key type lgssc registered [10239.425338] Lustre: Echo OBD driver; http://www.lustre.org/ [10280.716175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10280.742907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10282.026752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10282.062696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10282.147962] Lustre: lustre-MDT0000: new disk, initializing [10282.233377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10282.260580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10286.810605] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10298.796669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10298.937234] Lustre: 298945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10298.990178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10298.997752] Lustre: Skipped 1 previous similar message [10299.090235] Lustre: lustre-MDT0001: new disk, initializing [10299.185803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10299.256688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10299.273025] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10304.401566] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10315.299760] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10321.292732] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10328.663738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10329.182800] Lustre: lustre-OST0000: new disk, initializing [10329.191986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10329.201545] Lustre: 300912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10329.311614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10335.952529] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10336.853872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10336.868482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10336.996613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10345.267238] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10353.789885] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10363.838711] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10369.360847] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10408.931025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10408.931925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.937680] LustreError: Skipped 1 previous similar message [10408.954979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10408.965352] Lustre: Skipped 1 previous similar message [10408.990141] Lustre: Skipped 1 previous similar message [10414.063848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10414.072853] Lustre: Skipped 1 previous similar message [10419.174070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10419.181316] Lustre: Skipped 1 previous similar message [10421.216261] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10421.408312] Lustre: server umount lustre-OST0000 complete [10427.370337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10427.387719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.419268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10429.409432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10432.695609] Lustre: server umount lustre-MDT0000 complete [10436.753893] LustreError: 298937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849044 with bad export cookie 13606776163238843385 [10436.757717] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10436.771179] LustreError: 298937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10437.070640] Lustre: server umount lustre-MDT0001 complete [10448.746280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10454.443100] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10460.269090] Lustre: server umount MGS complete [10472.008223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10472.408376] LustreError: 304732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10472.423339] LustreError: 304732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10472.436850] LustreError: 304744:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10472.490688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10476.524031] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10477.540869] LustreError: 304731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10482.659479] LustreError: 304732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10485.508742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10485.894928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10490.033073] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10496.268638] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10500.767551] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10506.691808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10507.064816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10513.419034] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10521.244601] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10552.294574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10552.313716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10552.328775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10552.337006] Lustre: Skipped 2 previous similar messages [10558.309275] Lustre: server umount lustre-OST0000 complete [10562.539929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10562.547361] LustreError: Skipped 1 previous similar message [10562.557134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10562.568457] Lustre: Skipped 1 previous similar message [10562.572766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10562.576207] Lustre: Skipped 2 previous similar messages [10568.674577] LustreError: 305461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10568.692040] LustreError: 305461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10568.778102] Lustre: server umount lustre-MDT0000 complete [10572.875149] LustreError: 304711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849180 with bad export cookie 13606776163238845212 [10572.888402] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10572.894962] LustreError: 304711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10573.200496] Lustre: server umount lustre-MDT0001 complete [10586.028722] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [10589.045560] Key type lgssc unregistered [10589.354753] LNet: 308676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10589.366875] LNetError: 308676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10589.381068] LNet: Removed LNI 192.168.204.127@tcp [10590.615944] Key type .llcrypt unregistered [10590.618305] Key type ._llcrypt unregistered [10615.244382] Key type ._llcrypt registered [10615.251404] Key type .llcrypt registered [10615.391091] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [10630.944195] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10632.127019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10632.317733] alg: No test for adler32 (adler32-zlib) [10633.612376] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [10633.952602] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [10635.684395] Key type lgssc registered [10636.643439] Lustre: Echo OBD driver; http://www.lustre.org/ [10664.823271] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [10683.202330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10683.253468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.530755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10684.584927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10684.740424] Lustre: lustre-MDT0000: new disk, initializing [10685.003963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10685.042462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10690.697799] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [10703.806463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10704.177970] Lustre: lustre-OST0000: new disk, initializing [10704.185576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10704.189391] Lustre: Skipped 1 previous similar message [10704.196276] Lustre: 312726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10704.310388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10710.648753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10710.660763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10710.699900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10712.452911] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [11738.128303] Lustre: server umount lustre-MDT0000 complete [11742.705228] LustreError: 311913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850350 with bad export cookie 15463967193820716079 [11742.718609] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11753.122266] Lustre: server umount lustre-OST0000 complete [11782.337705] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [11793.967115] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [11834.637571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11834.877615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11834.908544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11834.971202] Lustre: lustre-MDT0000: new disk, initializing [11835.061561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11835.088426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11840.184524] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [11852.288885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11852.505368] Lustre: 317103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11852.620749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11852.628251] Lustre: Skipped 1 previous similar message [11852.742276] Lustre: lustre-MDT0001: new disk, initializing [11852.949610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11853.025744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11853.049942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11857.652665] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [11868.163377] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11874.427162] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11883.352267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11883.606290] Lustre: lustre-OST0000: new disk, initializing [11883.609718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11883.617842] Lustre: 319072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11883.731724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11889.210587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11889.225461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11889.338652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11890.398547] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [11900.506314] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11905.349857] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11905.562604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11911.258925] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11911.635551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11915.262922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11915.274431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11915.278453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11915.296911] Lustre: Skipped 1 previous similar message [11915.341878] LustreError: Skipped 1 previous similar message [11920.354786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11920.382814] Lustre: Skipped 2 previous similar messages [11920.719656] Lustre: server umount lustre-OST0000 complete [11925.495054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11925.505359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11925.510029] Lustre: Skipped 1 previous similar message [11925.543611] Lustre: Skipped 1 previous similar message [11930.600297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11930.616971] Lustre: Skipped 1 previous similar message [11931.892715] Lustre: server umount lustre-MDT0000 complete [11936.984398] LustreError: 317098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850544 with bad export cookie 15463967193823656898 [11937.018555] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11937.277123] Lustre: server umount lustre-MDT0001 complete [11942.044378] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11944.687870] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 13:09:10 (1787850550) [11952.983493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11953.679033] LustreError: 321387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11953.699192] LustreError: 321387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11953.773468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11958.760548] LustreError: 321388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11958.925233] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [11963.879150] LustreError: 321387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11968.220371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11968.592975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11973.152441] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [11981.017173] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11987.008526] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11994.635856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11995.017716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12001.413591] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12009.146726] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12016.812584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12017.033056] Lustre: lustre-OST0001: new disk, initializing [12017.038876] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12017.053379] Lustre: 324224:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12017.155881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12023.764567] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12027.105663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12027.122500] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12027.240856] LustreError: 324685:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12028.322833] LustreError: 324685:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12028.338373] LustreError: 324685:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [12029.351230] LustreError: 324690:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [12031.472223] LustreError: 324685:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12031.485860] LustreError: 324685:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [12034.694243] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12034.934190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12035.625678] LustreError: 324690:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [12035.650742] LustreError: 324690:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [12041.285670] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff995683218000.ost_server_uuid 50 [12043.139786] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff995683218000.ost_server_uuid in FULL state after 0 sec [12043.882311] LustreError: 324690:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [12043.894522] LustreError: 324690:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [12049.639277] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff995683218000.ost_server_uuid 50 [12051.214358] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff995683218000.ost_server_uuid in FULL state after 0 sec [12053.191312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12068.330442] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12068.340480] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12068.359247] Lustre: Skipped 1 previous similar message [12068.389483] Lustre: Skipped 1 previous similar message [12072.809570] Lustre: server umount lustre-OST0001 complete [12073.442307] LustreError: 323211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.465527] LustreError: 323211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12078.567611] LustreError: 323210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12078.587583] LustreError: 323210:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12081.126100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12081.146496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12081.160808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12081.170026] Lustre: Skipped 1 previous similar message [12086.845575] Lustre: server umount lustre-OST0000 complete [12091.361172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12091.373298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12091.393212] Lustre: Skipped 1 previous similar message [12097.873971] Lustre: server umount lustre-MDT0000 complete [12099.046349] LustreError: 321384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12099.074353] LustreError: 321384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12102.266127] LustreError: 321368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850710 with bad export cookie 15463967193823657773 [12102.267521] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12102.288035] LustreError: 321368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12102.698099] Lustre: server umount lustre-MDT0001 complete [12116.937497] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [12120.061818] Key type lgssc unregistered [12120.313734] LNet: 326915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12120.323664] LNetError: 326915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12120.343057] LNet: Removed LNI 192.168.204.127@tcp [12121.307943] Key type .llcrypt unregistered [12121.311163] Key type ._llcrypt unregistered [12156.137195] Key type ._llcrypt registered [12156.139286] Key type .llcrypt registered [12156.231647] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [12157.325898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12157.482860] alg: No test for adler32 (adler32-zlib) [12158.614136] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [12158.883492] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [12160.555056] Key type lgssc registered [12161.764754] Lustre: Echo OBD driver; http://www.lustre.org/ [12170.690638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12170.731628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.356816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12177.176451] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12182.498327] LustreError: 328336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12182.520895] LustreError: 328336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12186.358758] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12187.618571] LustreError: 328335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12192.741757] LustreError: 328336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.633750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12196.256208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12202.398627] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12215.584420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12216.390255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12217.389561] LustreError: 330018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12217.437892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [12221.558618] LustreError: 330019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12221.622654] LustreError: 330019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12227.838606] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12232.196822] LustreError: 330020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12232.242390] LustreError: 330020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12242.726566] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12252.658264] LustreError: 330019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.677135] LustreError: 330019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12255.338377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12255.852536] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12260.889683] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [12263.935594] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12273.106071] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12280.008304] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [12280.224731] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12393.957439] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12393.964305] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12394.001108] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12394.026949] Lustre: Skipped 1 previous similar message [12399.077979] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12399.101798] Lustre: Skipped 1 previous similar message [12404.209198] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12404.216836] Lustre: Skipped 1 previous similar message [12405.216281] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12405.446326] Lustre: server umount lustre-OST0001 complete [12409.323668] LustreError: 330020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12409.359205] LustreError: 330020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12414.449882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12414.468960] Lustre: Skipped 1 previous similar message [12414.485083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12414.495845] Lustre: Skipped 1 previous similar message [12419.518773] Lustre: server umount lustre-OST0000 complete [12426.732100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12426.750638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12426.772424] Lustre: Skipped 1 previous similar message [12426.782203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12429.798522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12431.383935] Lustre: server umount lustre-MDT0000 complete [12436.717938] LustreError: 332025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851044 with bad export cookie 6277978535281745280 [12436.724736] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12436.730353] LustreError: 332025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12437.183531] Lustre: server umount lustre-MDT0001 complete [12450.339441] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [12453.622180] Key type lgssc unregistered [12454.015592] LNet: 333286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12454.023290] LNetError: 333286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12454.038927] LNet: Removed LNI 192.168.204.127@tcp [12454.952804] Key type .llcrypt unregistered [12454.954796] Key type ._llcrypt unregistered [12511.179963] Key type ._llcrypt registered [12511.189475] Key type .llcrypt registered [12511.354672] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [12512.895457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12512.987393] alg: No test for adler32 (adler32-zlib) [12514.286562] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [12514.640842] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [12516.410828] Key type lgssc registered [12518.291369] Lustre: Echo OBD driver; http://www.lustre.org/ [12530.091137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12530.110863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12531.910818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12537.869307] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12541.419213] LustreError: 335575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12541.435920] LustreError: 335575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12546.531328] LustreError: 335576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.582166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.113402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12554.629076] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12558.251152] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12567.751939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12568.313769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12573.427075] LustreError: 337068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.511742] LustreError: 337066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12576.537635] LustreError: 337066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12576.540174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12577.042805] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12581.871405] LustreError: 337067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12581.907529] LustreError: 337067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12589.800310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12590.439664] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12595.781582] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12600.865843] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12611.989572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12617.498627] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12636.142471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12636.162874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12636.174661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12638.762765] Lustre: server umount lustre-MDT0000 complete [12641.768569] LustreError: 335570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12641.793337] LustreError: 335570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12643.152254] LustreError: 335557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851250 with bad export cookie 12249939016657351140 [12643.165434] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12643.181329] LustreError: 335557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12643.794589] Lustre: server umount lustre-MDT0001 complete [12660.721119] Lustre: server umount lustre-OST0000 complete [12662.560208] Lustre: 334837:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851254/real 1787851254] req@ffff9e0f4548f480 x1874697778173696/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787851270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12662.644844] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12662.664558] Lustre: Skipped 3 previous similar messages [12664.801675] Lustre: 334838:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851256/real 1787851256] req@ffff9e104d0c3800 x1874697778173952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787851272 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12666.708575] Lustre: server umount lustre-OST0001 complete [12692.962333] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [12706.245620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12706.749514] LustreError: 340846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12706.850982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12712.332190] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12723.205677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12723.682975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12729.032741] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12732.918618] Lustre: 341988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12741.762813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12742.452912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12748.600140] LustreError: 342345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12748.627402] LustreError: 342345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12751.688554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12751.939641] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12763.713322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12769.291534] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12771.527653] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [12782.207917] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12787.392829] Lustre: 343859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12794.850560] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12794.862600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12794.907517] Lustre: Skipped 2 previous similar messages [12794.922153] Lustre: Skipped 6 previous similar messages [12798.995120] Lustre: server umount lustre-MDT0000 complete [12804.450943] LustreError: 343963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851412 with bad export cookie 12249939016657352841 [12804.454283] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12804.457091] LustreError: 343963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12804.943333] Lustre: server umount lustre-MDT0001 complete [12819.477932] Lustre: server umount lustre-OST0000 complete [12820.320828] Lustre: 334839:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851412/real 1787851412] req@ffff9e10758ca300 x1874697778230272/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787851428 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12820.367030] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12820.397370] Lustre: Skipped 1 previous similar message [12824.546390] Lustre: 334837:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851416/real 1787851416] req@ffff9e107e912d80 x1874697778230528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787851432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12825.275895] Lustre: server umount lustre-OST0001 complete [12830.770645] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12833.279332] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12835.581784] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 13:24:01 (1787851441) [12864.570249] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [12874.969088] loop0: detected capacity change from 0 to 1099511627776 [12876.007932] SGI XFS with ACLs, security attributes, quota, no debug enabled [12876.028240] *** VALIDATE xfs *** [12876.058994] XFS (loop0): Mounting V5 Filesystem [12876.122696] XFS (loop0): Ending clean mount [12877.561906] loop1: detected capacity change from 0 to 18691697672192 [12998.864531] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [13011.433487] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [13058.420685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13058.733604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13058.764953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13058.860926] Lustre: lustre-MDT0000: new disk, initializing [13058.935693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13058.941782] Lustre: Skipped 1 previous similar message [13058.958913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13063.643377] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13077.114538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13077.357223] Lustre: 349014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13077.480242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13077.501260] Lustre: Skipped 1 previous similar message [13077.644508] Lustre: lustre-MDT0001: new disk, initializing [13077.998036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13078.031808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13085.423312] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13098.620811] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13106.208902] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13113.890084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13114.248208] Lustre: lustre-OST0000: new disk, initializing [13114.260731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13114.277218] Lustre: 350982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13115.945743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13115.967199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13116.120925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13122.294942] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13136.063849] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13143.202211] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13143.535258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13150.122294] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13150.418522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13157.344924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13157.349901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13157.359922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13157.375688] Lustre: Skipped 1 previous similar message [13159.264506] Lustre: server umount lustre-OST0000 complete [13165.025903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13165.040872] LustreError: Skipped 1 previous similar message [13165.050931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13165.060258] Lustre: Skipped 1 previous similar message [13167.590278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.608924] Lustre: Skipped 1 previous similar message [13170.515770] Lustre: server umount lustre-MDT0000 complete [13172.724536] LustreError: 350177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13172.752213] LustreError: 350177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [13174.874207] LustreError: 349817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851782 with bad export cookie 12249939016657354395 [13174.878184] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13174.888043] LustreError: 349817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13175.298657] Lustre: server umount lustre-MDT0001 complete [13185.962891] XFS (loop0): Unmounting Filesystem [13192.509886] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 13:29:58 (1787851798) [13200.938853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13201.652159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13201.663108] Lustre: Skipped 2 previous similar messages [13207.252549] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13218.638370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13225.907996] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13236.353328] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13243.718583] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13252.699315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13261.054207] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13270.785388] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13287.393464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13287.416173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13287.426680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13287.442747] Lustre: Skipped 1 previous similar message [13287.465057] Lustre: Skipped 3 previous similar messages [13292.289755] Lustre: server umount lustre-OST0000 complete [13297.639463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13304.211820] Lustre: server umount lustre-MDT0000 complete [13307.889826] LustreError: 353544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13307.909809] LustreError: 353544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13308.526569] LustreError: 353526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851916 with bad export cookie 12249939016657355277 [13308.542188] LustreError: 353526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13308.554436] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13309.011173] Lustre: server umount lustre-MDT0001 complete [13320.989636] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [13324.756635] Key type lgssc unregistered [13325.021549] LNet: 357179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13325.030550] LNetError: 357179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13325.057412] LNet: Removed LNI 192.168.204.127@tcp [13325.790482] Key type .llcrypt unregistered [13325.797686] Key type ._llcrypt unregistered [13364.434264] Key type ._llcrypt registered [13364.436272] Key type .llcrypt registered [13364.658577] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [13366.631268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13366.700376] alg: No test for adler32 (adler32-zlib) [13367.931931] Lustre: Lustre: Build Version: 2.17.57_85_g0128fd2 [13368.329648] LNet: Added LNI 192.168.204.127@tcp [8/256/0/180] [13370.064161] Key type lgssc registered [13371.752794] Lustre: Echo OBD driver; http://www.lustre.org/ [13382.163969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13382.254292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13384.130299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13390.788912] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13393.381756] LustreError: 358746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13393.410705] LustreError: 358746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13398.506556] LustreError: 358747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13402.248651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13402.814678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13408.251321] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13417.669822] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13425.162360] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13432.829233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13433.282203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13440.207947] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13449.477429] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13464.547464] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13464.557435] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13464.585118] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13465.093354] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13465.119705] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13465.131097] Lustre: Skipped 1 previous similar message [13470.182558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13470.202246] Lustre: Skipped 2 previous similar messages [13475.298128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13475.314846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13475.339711] Lustre: Skipped 2 previous similar messages [13475.606904] Lustre: server umount lustre-MDT0001 complete [13476.164317] LustreError: 358741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13480.422360] LustreError: 358742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13480.445043] LustreError: 358742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13485.538994] LustreError: 359476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13485.564328] LustreError: 359476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13487.896296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13488.026274] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13488.031672] Lustre: Found index 1 for lustre-MDT0001, updating log [13488.072864] Lustre: 358736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13488.465482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13493.616648] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13493.747093] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13497.351667] LustreError: 362355:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13497.370049] LustreError: 362355:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13497.385363] LustreError: 358738:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13497.587970] LustreError: 362358:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.127@tcp: cfg command failed: rc = -17 [13497.604332] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13497.604332] [13800.929928] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13800.960506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13800.965225] Lustre: Skipped 1 previous similar message [13808.973045] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [13808.984680] Lustre: Skipped 3 previous similar messages [13815.264133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13815.620939] Lustre: server umount lustre-MDT0001 complete [13816.290652] LustreError: 364046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13816.320554] LustreError: 364046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13834.576224] LustreError: 364046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13834.636676] LustreError: 364046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [13835.470951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13835.733224] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13835.755415] Lustre: Found index 1 for lustre-MDT0001, updating log [13835.872361] Lustre: 358735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13836.259062] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13841.403855] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13842.266254] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [13846.007372] LustreError: 370060:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13846.015237] LustreError: 370060:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13846.031355] LustreError: 358738:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13846.040140] LustreError: 358738:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13846.096944] LustreError: 370066:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13846.102091] LustreError: 370066:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13846.106324] LustreError: 370066:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.127@tcp: cfg command failed: rc = -17 [13846.113358] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.127@tcp [13846.113358] [14152.036196] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.27@tcp (stopping) [14152.045114] Lustre: Skipped 3 previous similar messages [14153.701661] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14153.717591] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14153.752165] Lustre: Skipped 1 previous similar message [14156.146427] Lustre: server umount lustre-MDT0001 complete [14157.149704] LustreError: 360182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14170.063797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14170.226267] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14170.232694] Lustre: Found index 1 for lustre-MDT0001, updating log [14170.366567] Lustre: 358734:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14170.993137] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [14176.244159] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14178.319691] LustreError: 377518:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [14178.346800] LustreError: 377518:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [14178.371792] LustreError: 358738:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [14178.390514] LustreError: 358738:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [14178.550029] LustreError: 377531:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [14178.630630] LustreError: 377531:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.127@tcp: cfg command failed: rc = -114 [14178.650374] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [14178.650374] [14179.023262] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [14503.913316] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14503.914508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14503.914540] Lustre: Skipped 2 previous similar messages [14503.916313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14503.974824] Lustre: Skipped 2 previous similar messages [14509.632932] Lustre: server umount lustre-MDT0000 complete [14513.771326] LustreError: 358727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853121 with bad export cookie 6735866643635796597 [14513.779552] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14513.793642] LustreError: 358727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14514.146142] LustreError: 358741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14514.150967] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14514.182515] LustreError: 358741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [14514.212781] Lustre: Skipped 1 previous similar message [14520.571366] Lustre: server umount lustre-MDT0001 complete [14526.962492] Lustre: server umount lustre-OST0000 complete [14540.991638] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [14554.277966] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [14598.277111] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 13:53:24 (1787853204) [14607.254920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14607.534292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14607.616835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14607.799378] Lustre: lustre-MDT0000: new disk, initializing [14607.974764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14608.030531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14613.413546] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [14625.226095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14625.384945] Lustre: 387491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14625.446842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14625.452502] Lustre: Skipped 1 previous similar message [14625.563747] Lustre: lustre-MDT0001: new disk, initializing [14625.692628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14625.727987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14625.744962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14630.901976] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [14641.353212] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14648.697558] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14657.232640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14657.651890] Lustre: lustre-OST0000: new disk, initializing [14657.655461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14657.662913] Lustre: 389462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14657.819106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14664.358868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14664.390820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14664.613680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14667.538226] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_default_debug -1 all [14680.715183] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14692.324986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14692.340225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14692.382443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14692.392080] Lustre: Skipped 7 previous similar messages [14696.205443] Lustre: server umount lustre-MDT0000 complete [14700.033985] LustreError: 387484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853307 with bad export cookie 6735866643635798550 [14700.040302] LustreError: MGC192.168.204.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14700.049945] LustreError: 387484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14700.543608] Lustre: server umount lustre-MDT0001 complete [14716.010044] Lustre: server umount lustre-OST0000 complete [14751.853784] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing set_hostid [14764.353716] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing load_modules_local [14802.624743] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14517 sec ========= 13:56:48 (1787853408) [14804.924979] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:56:50 (1787853410) === [14808.885619] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:56:54 (1787853414) === [14854.440961] Lustre: DEBUG MARKER: oleg427-server.virtnet: executing unload_modules_local [14857.537117] Key type lgssc unregistered [14857.887667] LNet: 395831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14857.896521] LNetError: 395831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14857.918565] LNet: Removed LNI 192.168.204.127@tcp [14858.925766] Key type .llcrypt unregistered [14858.929870] Key type ._llcrypt unregistered