[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 780697287 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2848136K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009140] LSM: Security Framework initializing [ 0.010056] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012066] *** VALIDATE selinux *** [ 0.022081] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026689] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] *** VALIDATE tmpfs *** [ 0.030457] *** VALIDATE proc *** [ 0.032234] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036074] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039027] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043095] debug: unmapping init [mem 0xffffffffafc59000-0xffffffffafc60fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046725] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050065] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054292] rcu: Hierarchical SRCU implementation. [ 0.056861] smp: Bringing up secondary CPUs ... [ 0.057583] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.064158] smp: Brought up 1 node, 4 CPUs [ 0.066014] smpboot: Max logical packages: 1 [ 0.067016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145026] node 0 deferred pages initialised in 76ms [ 0.153429] devtmpfs: initialized [ 0.155267] x86/mm: Memory block size: 128MB [ 0.163648] gcov: version magic: 0x41383552 [ 0.172328] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.177074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.181341] pinctrl core: initialized pinctrl subsystem [ 0.185188] [ 0.185854] ************************************************************* [ 0.191022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194013] ** ** [ 0.197013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.199011] ** ** [ 0.200012] ** This means that this kernel is built to expose internal ** [ 0.202013] ** IOMMU data structures, which may compromise security on ** [ 0.206030] ** your system. ** [ 0.208015] ** ** [ 0.210037] ** If you see this message and you are not debugging the ** [ 0.212013] ** kernel, report this immediately to your vendor! ** [ 0.215014] ** ** [ 0.219013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221012] ************************************************************* [ 0.224931] NET: Registered protocol family 16 [ 0.226453] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.228055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.230051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.234171] cpuidle: using governor menu [ 0.235816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.238604] PCI: Using configuration type 1 for base access [ 0.240156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.250090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.251019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.253112] cryptd: max_cpu_qlen set to 1000 [ 0.255220] ACPI: Added _OSI(Module Device) [ 0.256012] ACPI: Added _OSI(Processor Device) [ 0.257091] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.258014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.266242] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.275128] ACPI: Interpreter enabled [ 0.276070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.278018] ACPI: Using IOAPIC for interrupt routing [ 0.280104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.285824] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.304000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.308039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.310015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.315071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.322539] acpiphp: Slot [2] registered [ 0.324617] acpiphp: Slot [5] registered [ 0.326127] acpiphp: Slot [6] registered [ 0.328171] acpiphp: Slot [7] registered [ 0.330203] acpiphp: Slot [8] registered [ 0.332344] acpiphp: Slot [9] registered [ 0.334487] acpiphp: Slot [10] registered [ 0.336134] acpiphp: Slot [3] registered [ 0.338122] acpiphp: Slot [4] registered [ 0.341061] acpiphp: Slot [11] registered [ 0.343115] acpiphp: Slot [12] registered [ 0.345415] acpiphp: Slot [13] registered [ 0.348454] acpiphp: Slot [14] registered [ 0.350308] acpiphp: Slot [15] registered [ 0.353109] acpiphp: Slot [16] registered [ 0.355122] acpiphp: Slot [17] registered [ 0.358355] acpiphp: Slot [18] registered [ 0.360024] acpiphp: Slot [19] registered [ 0.361295] acpiphp: Slot [20] registered [ 0.363423] acpiphp: Slot [21] registered [ 0.366129] acpiphp: Slot [22] registered [ 0.368297] acpiphp: Slot [23] registered [ 0.371120] acpiphp: Slot [24] registered [ 0.373104] acpiphp: Slot [25] registered [ 0.375457] acpiphp: Slot [26] registered [ 0.378117] acpiphp: Slot [27] registered [ 0.380243] acpiphp: Slot [28] registered [ 0.382125] acpiphp: Slot [29] registered [ 0.384110] acpiphp: Slot [30] registered [ 0.385241] acpiphp: Slot [31] registered [ 0.387090] PCI host bridge to bus 0000:00 [ 0.390022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.394022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.398023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.402022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.405021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.409024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.411760] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.417105] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.421000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.437751] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.448060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.451017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.454016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.458017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.462588] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.467619] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.471039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.476905] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.486014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.519028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.531022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.546000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.558016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.579031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.611014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.632240] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.645016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.660013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.696016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.714427] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.729014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.744016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.769025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.789000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.803021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.817023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.853015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.871241] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.884015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.896014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.931017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.948000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.964031] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.979014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.009015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.033343] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.037877] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.041535] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.045444] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.049254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.055082] iommu: Default domain type: Passthrough [ 1.057416] SCSI subsystem initialized [ 1.059988] ACPI: bus type USB registered [ 1.061703] usbcore: registered new interface driver usbfs [ 1.065031] usbcore: registered new interface driver hub [ 1.068094] usbcore: registered new device driver usb [ 1.070164] pps_core: LinuxPPS API ver. 1 registered [ 1.074010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.080135] PTP clock support registered [ 1.084297] EDAC MC: Ver: 3.0.0 [ 1.086016] PCI: Using ACPI for IRQ routing [ 1.088000] NetLabel: Initializing [ 1.090011] NetLabel: domain hash size = 128 [ 1.093016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.096513] NetLabel: unlabeled traffic allowed by default [ 1.099614] vgaarb: loaded [ 1.102769] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.105019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.114338] clocksource: Switched to clocksource kvm-clock [ 1.302616] VFS: Disk quotas dquot_6.6.0 [ 1.304731] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.307112] *** VALIDATE ramfs *** [ 1.308950] *** VALIDATE hugetlbfs *** [ 1.310794] pnp: PnP ACPI init [ 1.313469] pnp: PnP ACPI: found 6 devices [ 1.330703] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.334962] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.337675] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.340089] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.342821] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.345566] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.348628] NET: Registered protocol family 2 [ 1.351180] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.356872] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.361220] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.366772] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.371054] TCP: Hash tables configured (established 65536 bind 65536) [ 1.374977] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.379769] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.384945] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.390643] NET: Registered protocol family 1 [ 1.396967] RPC: Registered named UNIX socket transport module. [ 1.401659] RPC: Registered udp transport module. [ 1.405512] RPC: Registered tcp transport module. [ 1.407944] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.413478] NET: Registered protocol family 44 [ 1.416564] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.420794] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.425422] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.429232] PCI: CLS 0 bytes, default 64 [ 1.432568] Unpacking initramfs... [ 4.336800] debug: unmapping init [mem 0xffff9f39bcc54000-0xffff9f39bffbffff] [ 4.343570] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.347379] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.352490] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.068676] Initialise system trusted keyrings [ 5.070706] Key type blacklist registered [ 5.073880] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.088597] zbud: loaded [ 5.093454] *** VALIDATE nfs *** [ 5.095363] *** VALIDATE nfs4 *** [ 5.097795] pstore: using deflate compression [ 5.103804] Platform Keyring initialized [ 5.282835] NET: Registered protocol family 38 [ 5.284516] Key type asymmetric registered [ 5.288117] Asymmetric key parser 'x509' registered [ 5.291341] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.296733] io scheduler mq-deadline registered [ 5.302856] io scheduler kyber registered [ 5.305888] io scheduler bfq registered [ 5.311412] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.316951] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.322177] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.327080] ACPI: Power Button [PWRF] [ 5.337194] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.349783] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.366034] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.393536] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.423353] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.466453] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.505591] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.514569] Non-volatile memory driver v1.3 [ 5.517877] Linux agpgart interface v0.103 [ 5.578585] virtio_blk virtio1: [vda] 146664 512-byte logical blocks (75.1 MB/71.6 MiB) [ 5.581639] vda: detected capacity change from 0 to 75091968 [ 5.603898] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.606835] vdb: detected capacity change from 0 to 1073741824 [ 5.626212] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.629223] vdc: detected capacity change from 0 to 2621440000 [ 5.646825] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.651533] vdd: detected capacity change from 0 to 2621440000 [ 5.670631] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.675172] vde: detected capacity change from 0 to 4294967296 [ 5.700672] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.705118] vdf: detected capacity change from 0 to 4294967296 [ 5.714471] libphy: Fixed MDIO Bus: probed [ 5.718966] usbcore: registered new interface driver usbserial_generic [ 5.722231] usbserial: USB Serial support registered for generic [ 5.724683] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.730893] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.734555] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.737246] mousedev: PS/2 mouse device common for all mice [ 5.742118] rtc_cmos 00:05: RTC can wake from S4 [ 5.746108] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.752505] rtc_cmos 00:05: registered as rtc0 [ 5.753984] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.759735] intel_pstate: CPU model not supported [ 5.764059] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.771815] hid: raw HID events driver (C) Jiri Kosina [ 5.777418] usbcore: registered new interface driver usbhid [ 5.780671] usbhid: USB HID core driver [ 5.782406] drop_monitor: Initializing network drop monitor service [ 5.785545] Initializing XFRM netlink socket [ 5.787835] NET: Registered protocol family 10 [ 5.788236] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.796797] Segment Routing with IPv6 [ 5.800547] NET: Registered protocol family 17 [ 5.802378] mpls_gso: MPLS GSO support [ 5.809232] RAS: Correctable Errors collector initialized. [ 5.811197] AVX version of gcm_enc/dec engaged. [ 5.812990] AES CTR mode by8 optimization enabled [ 5.929846] sched_clock: Marking stable (5929770929, 0)->(7264221582, -1334450653) [ 5.934921] registered taskstats version 1 [ 5.937493] Loading compiled-in X.509 certificates [ 5.939772] zswap: loaded using pool lzo/zbud [ 5.977240] Key type big_key registered [ 5.993896] Key type encrypted registered [ 5.998558] ima: No TPM chip found, activating TPM-bypass! [ 6.002817] ima: Allocated hash algorithm: sha1 [ 6.007337] ima: No architecture policies found [ 6.010214] evm: Initialising EVM extended attributes: [ 6.012736] evm: security.selinux [ 6.014603] evm: security.ima [ 6.015589] evm: security.capability [ 6.016941] evm: HMAC attrs: 0x1 [ 6.020730] rtc_cmos 00:05: setting system clock to 2026-09-08 01:55:14 UTC (1788832514) [ 6.030465] debug: unmapping init [mem 0xffffffffb0c03000-0xffffffffb0dfffff] [ 6.034789] debug: unmapping init [mem 0xffffffffaf982000-0xffffffffafc58fff] [ 6.044116] Write protecting the kernel read-only data: 28672k [ 6.047581] debug: unmapping init [mem 0xffffffffae003000-0xffffffffae1fffff] [ 6.050496] debug: unmapping init [mem 0xffffffffae914000-0xffffffffae9fffff] [ 6.098367] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.113487] systemd[1]: Detected virtualization kvm. [ 6.116711] systemd[1]: Detected architecture x86-64. [ 6.118708] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.152556] systemd[1]: No hostname configured. [ 6.154991] systemd[1]: Set hostname to . [ 6.156972] random: systemd: uninitialized urandom read (16 bytes read) [ 6.161751] systemd[1]: Initializing machine ID from random generator. [ 6.254938] random: ln: uninitialized urandom read (6 bytes read) [ 6.515412] random: systemd: uninitialized urandom read (16 bytes read) [ 6.519677] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.537701] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 6.554436] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.954580] device-mapper: uevent: version 1.0.3 [ 7.957473] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.527343] virtio_net virtio0 ens2: renamed from eth0 [ 9.670003] random: fast init done [ 9.941592] scsi host0: ata_piix [ 10.061636] scsi host1: ata_piix [ 10.062963] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.067778] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.527498] random: crng init done [ 15.528554] random: 7 urandom warning(s) missed due to ratelimiting [ 16.375882] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.672604] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.338634] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.801312] SELinux: Disabled at runtime. [ 20.893616] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.904610] systemd[1]: Detected virtualization kvm. [ 20.907555] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.106672] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.111279] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.120918] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.128723] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.139535] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.159406] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.178665] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 22.714762] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 23.775265] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.855288] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.222765] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.821815] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.950101] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.711369] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / n[ 31.527253] NFS: Registering the id_resolver key type [ 31.533547] Key type id_resolver registered o limit)[ 31.536698] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg619-server login: [ 45.159300] hrtimer: interrupt took 2672227 ns [ 106.193009] libcfs: loading out-of-tree module taints kernel. [ 106.248684] Key type ._llcrypt registered [ 106.250639] Key type .llcrypt registered [ 106.335512] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 123.320395] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 125.264554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 125.275562] alg: No test for adler32 (adler32-zlib) [ 126.863673] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 127.476840] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 129.232381] Key type lgssc registered [ 131.118205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.012321] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 184.912139] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 197.418671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 197.465330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 198.821298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 198.857573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.962914] Lustre: lustre-MDT0000: new disk, initializing [ 199.065958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.083583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 203.822966] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.198404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.312196] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.349827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.355360] Lustre: Skipped 1 previous similar message [ 216.504578] Lustre: lustre-MDT0001: new disk, initializing [ 216.613688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.674078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.693364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.590965] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.075713] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.461352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.719676] Lustre: lustre-OST0000: new disk, initializing [ 233.722596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 233.731319] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 233.774848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.647786] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.688477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.702869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.760222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 251.516586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.686993] Lustre: lustre-OST0001: new disk, initializing [ 251.697381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.701651] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.759069] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.718729] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.113382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 259.128615] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 259.214052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.237993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.413667] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 280.267737] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing check_logdir /tmp/testlogs/ [ 284.690166] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing yml_node [ 288.679162] Lustre: DEBUG MARKER: Client: 2.17.57.105 [ 290.934632] Lustre: DEBUG MARKER: MDS: 2.17.57.105 [ 293.909545] Lustre: DEBUG MARKER: OSS: 2.17.57.105 [ 295.546566] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 22:00:01 EDT 2026 [ 313.160841] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 314.793244] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 334.312329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 334.317458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 334.328572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.847555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.848244] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.880828] Lustre: Skipped 2 previous similar messages [ 337.320271] Lustre: server umount lustre-MDT0000 complete [ 340.928705] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832849 with bad export cookie 1699017448516989733 [ 340.933234] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.937231] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.971089] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.986405] Lustre: Skipped 1 previous similar message [ 340.990790] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.993396] Lustre: Skipped 3 previous similar messages [ 346.081549] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.081669] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.114640] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 346.142080] Lustre: Skipped 1 previous similar message [ 351.201340] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.201809] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 351.217481] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 351.223870] Lustre: Skipped 1 previous similar message [ 355.297049] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 355.530553] Lustre: server umount lustre-MDT0001 complete [ 359.131311] Lustre: server umount lustre-OST0000 complete [ 363.113529] Lustre: server umount lustre-OST0001 complete [ 370.214145] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 380.081191] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 411.440919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.762049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 411.819808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 411.911065] Lustre: lustre-MDT0000: new disk, initializing [ 411.992686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.015717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 416.827741] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 427.181624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.307368] Lustre: 15095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 427.356336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 427.363658] Lustre: Skipped 1 previous similar message [ 427.424224] Lustre: lustre-MDT0001: new disk, initializing [ 427.520707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 427.562357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 427.581376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 432.299537] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 442.281300] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 448.153769] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 454.631184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 454.923591] Lustre: lustre-OST0000: new disk, initializing [ 454.930488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 454.937924] Lustre: 17063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 455.060140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 456.244557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 456.252951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 456.368573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 460.535146] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 468.943112] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 472.943781] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 473.213237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 477.898584] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 478.130912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 482.277491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 482.281323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.284629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 482.289388] LustreError: Skipped 1 previous similar message [ 482.325422] Lustre: Skipped 1 previous similar message [ 494.048133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 494.255338] Lustre: server umount lustre-OST0000 complete [ 499.169318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.180176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.195322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 499.209428] Lustre: Skipped 5 previous similar messages [ 504.280641] Lustre: server umount lustre-MDT0000 complete [ 507.708132] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833016 with bad export cookie 1699017448516991259 [ 507.719185] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.721254] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 508.151411] Lustre: server umount lustre-MDT0001 complete [ 512.560860] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 22:03:38 (1788833018) [ 518.180990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.579593] LustreError: 19329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.665062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.278241] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 523.750426] LustreError: 19330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.866993] LustreError: 19329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.156521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.772599] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 542.996702] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 547.985168] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 554.519629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 561.357909] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 569.491362] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 573.234519] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 579.559897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.573656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 579.574808] Lustre: Skipped 2 previous similar messages [ 579.584883] Lustre: Skipped 2 previous similar messages [ 585.976777] Lustre: server umount lustre-MDT0000 complete [ 589.653871] LustreError: 19310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833098 with bad export cookie 1699017448516992148 [ 589.660994] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 589.667209] LustreError: 19310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 589.793831] LustreError: 21494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.916490] Lustre: server umount lustre-MDT0001 complete [ 595.928295] Lustre: server umount lustre-OST0000 complete [ 615.221942] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 624.187629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.722203] LustreError: 23707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.737584] LustreError: 23707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 624.775754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.782271] Lustre: Skipped 2 previous similar messages [ 629.599912] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 638.579623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.286624] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 646.012554] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 652.848100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 659.013352] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 666.670595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 666.859230] Lustre: lustre-OST0001: new disk, initializing [ 666.872374] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 666.884586] Lustre: 26023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 668.375621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 668.393380] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 668.492446] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 672.824903] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 682.186668] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 685.727092] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 696.303378] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.313765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 696.334769] Lustre: Skipped 5 previous similar messages [ 696.345396] Lustre: Skipped 6 previous similar messages [ 701.719991] Lustre: server umount lustre-MDT0000 complete [ 705.431620] LustreError: 26022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833213 with bad export cookie 1699017448516992995 [ 705.434960] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 705.439850] LustreError: 26022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 705.673549] Lustre: server umount lustre-MDT0001 complete [ 719.562960] Lustre: server umount lustre-OST0000 complete [ 733.296537] Lustre: server umount lustre-OST0001 complete [ 749.497970] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 760.048895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.450205] LustreError: 28993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 760.469491] LustreError: 28993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 760.531197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.537619] Lustre: Skipped 3 previous similar messages [ 764.914868] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 773.080144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.592260] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 780.322463] Lustre: 30139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 786.837294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 792.661435] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 800.750216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 807.663154] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 814.789594] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 817.742225] Lustre: 32019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 824.802209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 824.812399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 824.832770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 824.841223] Lustre: Skipped 4 previous similar messages [ 828.690824] Lustre: server umount lustre-MDT0000 complete [ 830.949156] LustreError: 29727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 830.969058] LustreError: 29727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 832.007023] LustreError: 30521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833340 with bad export cookie 1699017448516994290 [ 832.017859] LustreError: 30521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 832.021166] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.431865] Lustre: server umount lustre-MDT0001 complete [ 846.405288] Lustre: server umount lustre-OST0000 complete [ 861.063535] Lustre: server umount lustre-OST0001 complete [ 869.919672] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 22:09:35 (1788833375) [ 894.905318] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 904.103324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.978928] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 917.072080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.743071] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 924.311289] Lustre: 35972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 930.195520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 937.140624] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 945.073801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 951.225510] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 958.772403] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 962.252402] Lustre: 37852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 968.677323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 968.687370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.701789] Lustre: Skipped 3 previous similar messages [ 973.380294] Lustre: server umount lustre-MDT0000 complete [ 975.335593] LustreError: 34827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.361660] LustreError: 34827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 976.538070] LustreError: 34807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833485 with bad export cookie 1699017448516995396 [ 976.547695] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.854600] Lustre: server umount lustre-MDT0001 complete [ 989.752942] Lustre: server umount lustre-OST0000 complete [ 1004.053372] Lustre: server umount lustre-OST0001 complete [ 1011.874684] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 22:11:57 (1788833517) [ 1026.855497] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1036.431747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.961622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.970385] Lustre: Skipped 7 previous similar messages [ 1041.759552] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1050.193638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.902482] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1057.953806] Lustre: 41148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1063.817507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1070.123949] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1077.648307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1083.967730] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1091.274480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1094.229290] Lustre: 43024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1094.953533] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1094.960303] Lustre: Skipped 1 previous similar message [ 1099.236741] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.256846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1099.260595] Lustre: Skipped 6 previous similar messages [ 1099.280545] Lustre: Skipped 12 previous similar messages [ 1103.878708] Lustre: server umount lustre-MDT0000 complete [ 1107.503625] LustreError: 39977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833615 with bad export cookie 1699017448516996502 [ 1107.508876] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.513867] LustreError: 39977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1113.906574] Lustre: server umount lustre-MDT0001 complete [ 1121.524533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1133.034193] LustreError: 3651:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f3a3d5b5c00 x1875726969463296/t0(0) o250->MGC192.168.206.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1137.583587] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1142.567725] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1145.356052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.264807] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1150.970349] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1150.975675] Lustre: Skipped 1 previous similar message [ 1157.630848] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1159.002197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1164.358602] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1166.211385] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1171.441580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1177.259821] Lustre: server umount lustre-MDT0000 complete [ 1180.348314] LustreError: 39975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833688 with bad export cookie 1699017448516997629 [ 1180.356427] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1180.371818] LustreError: 39975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1180.843691] Lustre: server umount lustre-MDT0001 complete [ 1194.978396] Lustre: server umount lustre-OST0000 complete [ 1197.796548] Lustre: 3654:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788833690/real 1788833690] req@ffff9f3a3d101500 x1875726969496192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788833706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1198.896995] Lustre: server umount lustre-OST0001 complete [ 1207.731170] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 22:15:13 (1788833713) [ 1222.634743] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1231.311624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.622744] LustreError: 47626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1231.637934] LustreError: 47626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 1235.806601] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1243.731934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.435743] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1251.237677] Lustre: 48777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1257.913894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1264.666519] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1272.038192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1278.041847] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1284.390393] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1288.547778] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1300.398353] Lustre: server umount lustre-MDT0000 complete [ 1303.789591] LustreError: 47607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833812 with bad export cookie 1699017448516998427 [ 1303.792093] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.798615] LustreError: 47607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1304.144550] Lustre: server umount lustre-MDT0001 complete [ 1307.974606] Lustre: server umount lustre-OST0000 complete [ 1311.705336] Lustre: server umount lustre-OST0001 complete [ 1320.807409] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 22:17:06 (1788833826) [ 1323.093989] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1324.785286] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 22:17:10 (1788833830) [ 1334.177947] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 1341.699981] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1363.046239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.322047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1363.407957] Lustre: lustre-MDT0000: new disk, initializing [ 1363.488656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1367.697570] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1376.442857] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1382.837438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1383.040464] Lustre: lustre-OST0000: new disk, initializing [ 1383.044623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1383.049955] Lustre: Skipped 1 previous similar message [ 1383.056521] Lustre: 54849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1385.074204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1385.082945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1385.111534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1389.086916] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1398.014690] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1400.114706] Lustre: DEBUG MARKER: create test files [ 1425.891461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.900713] Lustre: Skipped 13 previous similar messages [ 1430.772236] Lustre: server umount lustre-OST0000 complete [ 1433.825538] Lustre: server umount lustre-MDT0000 complete [ 1442.146804] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 1444.503247] Key type lgssc unregistered [ 1444.767411] LNet: 56478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.779732] LNetError: 56478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.796817] LNet: Removed LNI 192.168.206.119@tcp [ 1445.728236] Key type .llcrypt unregistered [ 1445.731489] Key type ._llcrypt unregistered [ 1468.908256] Key type ._llcrypt registered [ 1468.910503] Key type .llcrypt registered [ 1469.019392] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1469.751787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.767581] alg: No test for adler32 (adler32-zlib) [ 1470.820159] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 1471.113374] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 1472.776124] Key type lgssc registered [ 1474.181888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1480.931657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1480.968699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.295803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1485.944977] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1491.700830] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1497.059683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.296905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.964293] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1506.540151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 1506.620918] Lustre: 59183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1506.626374] Lustre: 59183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1506.630928] Lustre: 59183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1506.636461] Lustre: 59183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1506.641459] Lustre: 59183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1506.647722] Lustre: 59183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1508.314731] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1512.831120] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1517.092392] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1552.871551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1552.878057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.895732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1555.166648] Lustre: server umount lustre-OST0000 complete [ 1558.624151] Lustre: server umount lustre-MDT0000 complete [ 1568.982857] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 1571.990947] Key type lgssc unregistered [ 1572.301429] LNet: 60814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.312827] LNetError: 60814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.341774] LNet: Removed LNI 192.168.206.119@tcp [ 1573.114956] Key type .llcrypt unregistered [ 1573.118531] Key type ._llcrypt unregistered [ 1599.678740] Key type ._llcrypt registered [ 1599.680704] Key type .llcrypt registered [ 1599.773095] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1600.625994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1600.711955] alg: No test for adler32 (adler32-zlib) [ 1601.768662] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 1601.998787] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 1603.657407] Key type lgssc registered [ 1604.713362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1610.922846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1610.946209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1612.250759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.526459] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1622.986157] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1628.515318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1628.761484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1633.820971] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1636.973914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1637.062385] Lustre: 63622:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1637.069311] Lustre: 63622:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1637.079481] Lustre: 63622:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1637.088441] Lustre: 63622:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1637.095617] Lustre: 63622:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1637.103388] Lustre: 63622:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1639.870981] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1644.556759] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1652.705384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1652.713246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.724063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1657.612616] Lustre: server umount lustre-OST0000 complete [ 1661.420525] Lustre: server umount lustre-MDT0000 complete [ 1671.891521] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 1674.712523] Key type lgssc unregistered [ 1675.068503] LNet: 65178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.082968] LNetError: 65178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.096806] LNet: Removed LNI 192.168.206.119@tcp [ 1676.062967] Key type .llcrypt unregistered [ 1676.065361] Key type ._llcrypt unregistered [ 1693.932343] Key type ._llcrypt registered [ 1693.935842] Key type .llcrypt registered [ 1694.024859] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 1707.350154] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1708.207374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1708.368864] alg: No test for adler32 (adler32-zlib) [ 1709.401685] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 1709.682756] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 1711.384583] Key type lgssc registered [ 1712.658816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1744.861895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1744.907134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.158727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1746.195401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1746.300405] Lustre: lustre-MDT0000: new disk, initializing [ 1746.384837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.410960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1750.423401] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1760.350700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.477962] Lustre: 68327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1760.535111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1760.539132] Lustre: Skipped 1 previous similar message [ 1760.596388] Lustre: lustre-MDT0001: new disk, initializing [ 1760.692879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1760.752888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1760.769054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1764.906976] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1773.867984] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1779.411265] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1786.104848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1786.476348] Lustre: lustre-OST0000: new disk, initializing [ 1786.485704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1786.494975] Lustre: 70295:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1786.595332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.908748] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1794.130124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1794.142135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1794.303706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1800.112929] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1804.897507] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1805.100977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1809.715631] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1809.909855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1815.011233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1815.013634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.027270] LustreError: Skipped 1 previous similar message [ 1815.032461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1815.040454] Lustre: Skipped 1 previous similar message [ 1815.051869] Lustre: Skipped 1 previous similar message [ 1817.900727] Lustre: server umount lustre-OST0000 complete [ 1822.178617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1822.187561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.208461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.250680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1825.256779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.401208] Lustre: server umount lustre-MDT0000 complete [ 1830.892314] LustreError: 68318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834339 with bad export cookie 4718444021218641053 [ 1830.901272] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.911037] LustreError: 68318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1831.239613] Lustre: server umount lustre-MDT0001 complete [ 1839.478438] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 22:25:45 (1788834345) [ 1849.924698] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1884.869353] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 1895.817545] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 1928.416391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1928.645361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1928.676864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1928.779306] Lustre: lustre-MDT0000: new disk, initializing [ 1928.852444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.868281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1933.095234] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1941.770883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.879152] Lustre: 76314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1941.985153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1941.991384] Lustre: Skipped 1 previous similar message [ 1942.119976] Lustre: lustre-MDT0001: new disk, initializing [ 1942.206259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1942.241725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1942.253790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1946.129443] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1954.917829] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1960.407202] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1967.134382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.420544] Lustre: lustre-OST0000: new disk, initializing [ 1967.423407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1967.428136] Lustre: 78280:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1967.548532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.889364] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 1974.911421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1974.929457] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1975.104350] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1983.545220] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1987.868996] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1988.027289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1991.978668] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1992.203195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1995.750175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1995.751993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.765848] LustreError: Skipped 1 previous similar message [ 1995.780133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.803146] Lustre: Skipped 2 previous similar messages [ 2000.133285] Lustre: server umount lustre-OST0000 complete [ 2005.992416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.000245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.004895] Lustre: Skipped 2 previous similar messages [ 2006.013833] Lustre: Skipped 1 previous similar message [ 2016.225461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.232368] Lustre: Skipped 3 previous similar messages [ 2018.272128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2018.382268] LustreError: 79820:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000a92d858c) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2018.543829] Lustre: server umount lustre-MDT0000 complete [ 2021.346804] LustreError: 76323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.363233] LustreError: 76323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2022.416070] LustreError: 76304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834530 with bad export cookie 4718444021218641935 [ 2022.416864] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2022.443899] LustreError: 76304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2022.815846] Lustre: server umount lustre-MDT0001 complete [ 2031.386101] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 22:28:57 (1788834537) [ 2037.815719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.326096] LustreError: 80693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.415806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.140587] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2043.877645] LustreError: 80694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2047.970959] LustreError: 80693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2050.841380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.453464] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2060.857265] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.703208] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2071.453037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2071.786815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.793852] Lustre: Skipped 1 previous similar message [ 2077.933779] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2085.526896] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2088.471268] Lustre: 83362:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 40476fb4-fe25-4aca-90b6-e3360d74203b at adminstrative request [ 2093.538988] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2093.547441] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2103.783557] Lustre: MGS: Client 40476fb4-fe25-4aca-90b6-e3360d74203b (at 0@lo) reconnecting [ 2103.804449] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x417b47d6814d8b81 to 0x417b47d6814d8e21 [ 2103.820180] Lustre: MGC192.168.206.119@tcp: Connection restored to 0@lo (at 0@lo) [ 2124.938497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2125.152333] Lustre: lustre-OST0001: new disk, initializing [ 2125.157671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2125.166716] Lustre: 83755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2131.085255] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2135.145618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2135.165013] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2135.289952] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2140.424332] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2145.763374] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2145.764928] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.771742] LustreError: Skipped 1 previous similar message [ 2145.773890] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2145.782350] Lustre: Skipped 1 previous similar message [ 2145.793933] Lustre: Skipped 1 previous similar message [ 2149.393938] Lustre: server umount lustre-OST0001 complete [ 2150.901414] LustreError: 82513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.917512] LustreError: 82513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2152.929194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2159.015976] LustreError: 82510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.041852] LustreError: 82510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2167.264734] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2167.396278] Lustre: server umount lustre-OST0000 complete [ 2174.433159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2174.438738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.441137] LustreError: Skipped 1 previous similar message [ 2174.454174] Lustre: Skipped 3 previous similar messages [ 2177.417798] Lustre: server umount lustre-MDT0000 complete [ 2179.553880] LustreError: 80694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.575143] LustreError: 80694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2181.002183] LustreError: 80675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834689 with bad export cookie 4718444021218643489 [ 2181.009793] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2181.013365] LustreError: 80675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2181.367304] Lustre: server umount lustre-MDT0001 complete [ 2188.682131] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 22:31:34 (1788834694) [ 2190.028905] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2191.716280] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 22:31:37 (1788834697) [ 2193.103188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2194.831658] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 22:31:40 (1788834700) [ 2196.204173] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2197.569860] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 22:31:43 (1788834703) [ 2198.824744] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2200.408724] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 22:31:46 (1788834706) [ 2201.590323] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2206.547929] loop: module loaded [ 2206.565358] loop0: detected capacity change from 0 to 4294967296 [ 2206.619919] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2208.117137] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2211.807875] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2214.855788] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2239.412781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2239.475723] LustreError: 87881:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2239.484245] LustreError: 87881:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2239.502939] LustreError: 87881:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2239.522690] LustreError: 87881:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2256.519235] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 2266.447408] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 2302.945402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.186803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2303.218259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2303.302429] Lustre: lustre-MDT0000: new disk, initializing [ 2303.408226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.415606] Lustre: Skipped 1 previous similar message [ 2303.435594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2308.182896] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2318.360125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.473450] Lustre: 90745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2318.731867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2322.601404] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2331.613410] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2337.135953] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2343.721417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2343.975975] Lustre: 92713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2349.641232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2349.753142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2351.244869] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2360.246546] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2364.685876] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2364.934300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2368.938314] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2369.198457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2375.655931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.676165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2375.683775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2375.685401] Lustre: Skipped 11 previous similar messages [ 2375.699491] LustreError: Skipped 1 previous similar message [ 2385.376829] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2385.517056] Lustre: server umount lustre-OST0000 complete [ 2403.296353] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2403.608342] Lustre: server umount lustre-MDT0000 complete [ 2406.371864] LustreError: 93239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.407377] LustreError: 93239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2407.454592] LustreError: 90739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834915 with bad export cookie 4718444021218644105 [ 2407.463252] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2407.474863] LustreError: 90739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2407.796860] Lustre: server umount lustre-MDT0001 complete [ 2416.164647] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 22:35:21 (1788834921) [ 2422.789107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.236626] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2436.979621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.281519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2437.286861] Lustre: Skipped 3 previous similar messages [ 2442.116125] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2448.953509] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2453.783702] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2460.103126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2466.759323] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2473.580424] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2478.606680] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2478.811417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2483.112878] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2483.318558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2489.617780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2489.795958] Lustre: lustre-OST0001: new disk, initializing [ 2489.799934] Lustre: Skipped 2 previous similar messages [ 2489.803536] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2489.808374] Lustre: Skipped 3 previous similar messages [ 2489.812986] Lustre: 98498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2496.809901] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2497.163285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2497.172305] Lustre: Skipped 2 previous similar messages [ 2497.179072] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2497.290382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2506.382891] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2510.624388] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2510.873207] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2515.852712] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2516.081717] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2534.485617] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2556.806426] Lustre: Failing over lustre-MDT0000 [ 2557.358343] Lustre: server umount lustre-MDT0000 complete [ 2558.960034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.964857] LustreError: 95125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.979886] Lustre: Skipped 4 previous similar messages [ 2559.025102] LustreError: 95125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2575.336305] Lustre: 66552:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788835067/real 1788835067] req@00000000ec2eeed1 x1875728629243008/t0(0) o400->MGC192.168.206.119@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788835083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2575.392893] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2580.530198] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2580.533193] LDISKFS-fs (dm-0): recovery complete [ 2580.542505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.547111] LustreError: 66549:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007045d37b x1875728629251456/t0(0) o250->MGC192.168.206.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2584.818729] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2588.463346] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2589.921606] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2589.923324] Lustre: 95124:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2590.203455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2590.275857] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2590.600120] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2590.609303] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2590.621042] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2590.946468] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2590.959860] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2591.281587] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2591.289192] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2591.619513] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2591.629532] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 2 previous similar messages [ 2591.955396] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2591.966814] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2592.288946] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2592.296754] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2593.897215] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2593.917202] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 2594.234528] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2594.243359] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2594.560327] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2594.569824] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2598.049706] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2598.061548] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2598.385286] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2598.398750] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2598.720113] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2598.725132] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2606.323057] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2606.335524] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 25 previous similar messages [ 2606.657155] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2606.674899] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 25 previous similar messages [ 2607.002775] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2607.011213] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 25 previous similar messages [ 2622.523779] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2622.528518] Lustre: 101355:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 2622.862296] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2622.880341] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2623.208117] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2623.212294] LustreError: 101355:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2649.176757] Lustre: 101355:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2649.187720] Lustre: 101355:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2649.200745] Lustre: 101355:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 808b9365-c0a6-4a28-adec-de5b6e051684@192.168.206.19@tcp [ 2649.217483] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2649.223945] Lustre: 101355:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@0000000043dc6001 x1875728614364416/t0(8589934782) o35->808b9365-c0a6-4a28-adec-de5b6e051684@192.168.206.19@tcp:78/0 lens 392/0 e 0 to 0 dl 1788835168 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2649.256684] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2649.274418] Lustre: lustre-MDT0000: Denying connection for new client 808b9365-c0a6-4a28-adec-de5b6e051684 (at 192.168.206.19@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2649.288511] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2649.339360] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2649.351123] Lustre: Skipped 2 previous similar messages [ 2649.352546] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2649.393797] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:49 to 0x2c0000401:545) [ 2649.398946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2665.441146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2665.448601] LustreError: Skipped 1 previous similar message [ 2665.453534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2665.465180] Lustre: Skipped 9 previous similar messages [ 2668.900788] Lustre: server umount lustre-OST0000 complete [ 2678.837808] Lustre: server umount lustre-OST0001 complete [ 2688.276269] Lustre: server umount lustre-MDT0000 complete [ 2692.560504] LustreError: 95110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835201 with bad export cookie 4718444021218695317 [ 2692.563426] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2692.570546] LustreError: 95110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2693.164807] Lustre: server umount lustre-MDT0001 complete [ 2701.696912] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 22:40:07 (1788835207) [ 2707.995242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.006872] Lustre: *** cfs_fail_loc=197, val=0*** [ 2723.616675] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2728.736499] LustreError: 103331:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.119@tcp: failed processing log, type 4: rc = -110 [ 2733.965537] Lustre: *** cfs_fail_loc=197, val=0*** [ 2733.969375] Lustre: Skipped 65 previous similar messages [ 2754.528329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.541644] Lustre: Skipped 3 previous similar messages [ 2761.383506] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2769.236497] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2771.004757] Lustre: server umount lustre-OST0000 complete [ 2778.661772] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 22:41:24 (1788835284) [ 2798.420168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.718787] LustreError: 104951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.729351] LustreError: 104951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 2802.883896] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2811.771122] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2815.729443] LustreError: 104982:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2815.744143] LustreError: 104982:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2815.757194] LustreError: 104982:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2816.161763] Lustre: server umount lustre-MDT0000 complete [ 2850.257473] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 2860.317788] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 2897.538117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2897.716965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2897.744831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2897.839810] Lustre: lustre-MDT0000: new disk, initializing [ 2897.936468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2902.966603] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2914.518819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.670294] Lustre: 109159:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2914.764709] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2914.773065] Lustre: Skipped 10 previous similar messages [ 2915.055882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2920.550746] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2931.632389] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2938.308566] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2945.821959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2946.134117] Lustre: 111128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2952.888106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2953.751106] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 2963.155833] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2967.942511] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2968.218836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2973.634847] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2973.926804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2978.790268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2978.790736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2978.813944] Lustre: Skipped 8 previous similar messages [ 2978.820176] LustreError: Skipped 4 previous similar messages [ 2982.427979] Lustre: server umount lustre-OST0000 complete [ 3001.313895] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3001.771859] Lustre: server umount lustre-MDT0000 complete [ 3006.221053] LustreError: 109149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835514 with bad export cookie 4718444021218706041 [ 3006.230720] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3006.236635] LustreError: 109149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3006.605482] Lustre: server umount lustre-MDT0001 complete [ 3016.734352] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 22:45:22 (1788835522) [ 3025.892293] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 3030.172234] Key type lgssc unregistered [ 3030.759833] LNet: 113615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3030.781571] LNetError: 113615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3030.830315] LNet: Removed LNI 192.168.206.119@tcp [ 3032.004954] Key type .llcrypt unregistered [ 3032.006605] Key type ._llcrypt unregistered [ 3053.335989] Key type ._llcrypt registered [ 3053.338184] Key type .llcrypt registered [ 3053.409419] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 3068.601969] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 3069.749401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3069.778276] alg: No test for adler32 (adler32-zlib) [ 3070.920774] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 3071.228423] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 3072.952217] Key type lgssc registered [ 3074.415792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3114.436087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3114.499504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.822791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3115.847168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3115.922584] Lustre: lustre-MDT0000: new disk, initializing [ 3116.119339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.161559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3120.942765] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3132.399280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.885186] Lustre: lustre-OST0000: new disk, initializing [ 3132.894734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3132.900234] Lustre: Skipped 1 previous similar message [ 3132.907442] Lustre: 117784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3133.053334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.399231] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3142.797621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3142.818587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3142.870353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3151.014785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.322257] Lustre: lustre-OST0001: new disk, initializing [ 3151.326649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3151.332346] Lustre: 118653:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3151.393356] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3156.548845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3156.564778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3156.616226] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3158.246344] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3171.810036] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.837256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3171.845742] Lustre: Skipped 1 previous similar message [ 3173.834472] LustreError: 119258:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000003d04a92a) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3174.101155] Lustre: server umount lustre-MDT0000 complete [ 3180.944051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3186.855502] LustreError: 116973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835695 with bad export cookie 10981281160160660353 [ 3186.870099] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3186.882457] LustreError: 116973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3187.203818] Lustre: server umount lustre-OST0000 complete [ 3191.407448] Lustre: server umount lustre-OST0001 complete [ 3200.333436] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 22:48:25 (1788835705) [ 3223.286911] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 22:48:49 (1788835729) [ 3234.645436] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 3259.978927] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 22:49:25 (1788835765) [ 3299.855784] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 3309.610347] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 3347.289332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.641848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3347.688818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3347.794870] Lustre: lustre-MDT0000: new disk, initializing [ 3347.849851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3347.878716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3352.502511] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3363.655768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3363.791282] Lustre: 126403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3363.849865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3363.855656] Lustre: Skipped 1 previous similar message [ 3363.937258] Lustre: lustre-MDT0001: new disk, initializing [ 3364.064715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3364.115495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3364.132110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3368.198184] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3376.852562] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3382.386226] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3388.556624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.013831] Lustre: lustre-OST0000: new disk, initializing [ 3389.026712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3389.038265] Lustre: 128368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3389.233357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.284881] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3396.729202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3396.741321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3396.853471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3407.466634] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3412.374162] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3412.783975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3417.640668] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3417.914986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3422.692146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3422.693093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3422.696756] LustreError: Skipped 1 previous similar message [ 3422.699666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3422.700943] Lustre: Skipped 2 previous similar messages [ 3425.814227] Lustre: server umount lustre-OST0000 complete [ 3430.882598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3430.886362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3430.896039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.898421] Lustre: Skipped 1 previous similar message [ 3432.929803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.930926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3435.679461] Lustre: server umount lustre-MDT0000 complete [ 3438.050743] LustreError: 126413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.076574] LustreError: 126413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3439.374413] LustreError: 126393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835947 with bad export cookie 10981281160160661571 [ 3439.378423] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3439.384553] LustreError: 126393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3439.761281] Lustre: server umount lustre-MDT0001 complete [ 3449.041961] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 22:52:34 (1788835954) [ 3456.615035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3457.174982] LustreError: 130783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.272436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.627855] LustreError: 130784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.667388] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3466.728249] LustreError: 130783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.326374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.408671] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3483.816934] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3489.665922] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3495.869101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3496.276593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3496.280636] Lustre: Skipped 1 previous similar message [ 3502.118358] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3509.930269] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3514.329983] Lustre: *** cfs_fail_loc=159, val=0*** [ 3530.007945] Lustre: lustre-MDT0000: Client a8691e8f-2b41-4317-9eaa-a6ac6413b3b7 (at 192.168.206.19@tcp) reconnecting [ 3531.350633] Lustre: *** cfs_fail_loc=159, val=0*** [ 3531.354107] Lustre: Skipped 5 previous similar messages [ 3546.916690] Lustre: lustre-MDT0000: Client a8691e8f-2b41-4317-9eaa-a6ac6413b3b7 (at 192.168.206.19@tcp) reconnecting [ 3553.764390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.767351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.772789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.772796] Lustre: Skipped 1 previous similar message [ 3553.786553] Lustre: Skipped 1 previous similar message [ 3558.499564] Lustre: server umount lustre-OST0000 complete [ 3564.002152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3564.002994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3564.025891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3564.027848] Lustre: Skipped 1 previous similar message [ 3568.781567] Lustre: server umount lustre-MDT0000 complete [ 3569.123714] LustreError: 130784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.136441] LustreError: 130784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3572.302453] LustreError: 133501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836080 with bad export cookie 10981281160160662460 [ 3572.309027] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3572.312713] LustreError: 133501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3572.606785] Lustre: server umount lustre-MDT0001 complete [ 3582.408579] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 3584.873869] Key type lgssc unregistered [ 3585.171877] LNet: 134554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3585.195883] LNetError: 134554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3585.239871] LNet: Removed LNI 192.168.206.119@tcp [ 3586.024818] Key type .llcrypt unregistered [ 3586.030462] Key type ._llcrypt unregistered [ 3611.776780] Key type ._llcrypt registered [ 3611.780755] Key type .llcrypt registered [ 3611.890419] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 3612.920365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3612.981493] alg: No test for adler32 (adler32-zlib) [ 3614.108954] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 3614.322035] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 3615.976407] Key type lgssc registered [ 3617.373102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3624.995218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3625.046868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.668466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3631.521873] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3637.222121] LustreError: 135965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.240875] LustreError: 135965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3639.655730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.069853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3644.605976] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3651.792157] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3656.457629] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3663.184925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3663.501506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.301957] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3671.730083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3678.832961] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3688.515344] Lustre: *** cfs_fail_loc=159, val=0*** [ 3704.097604] Lustre: lustre-MDT0000: Client f76a02aa-3a17-4879-8522-24d1d3b762d8 (at 192.168.206.19@tcp) reconnecting [ 3706.098198] Lustre: *** cfs_fail_loc=159, val=0*** [ 3722.518874] Lustre: lustre-MDT0001: Client f76a02aa-3a17-4879-8522-24d1d3b762d8 (at 192.168.206.19@tcp) reconnecting [ 3723.895477] Lustre: *** cfs_fail_loc=159, val=0*** [ 3723.897354] Lustre: Skipped 3 previous similar messages [ 3740.463962] Lustre: lustre-MDT0001: Client f76a02aa-3a17-4879-8522-24d1d3b762d8 (at 192.168.206.19@tcp) reconnecting [ 3745.694516] Lustre: 139002:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3748.296298] Lustre: *** cfs_fail_loc=159, val=0*** [ 3748.301893] Lustre: Skipped 4 previous similar messages [ 3763.971587] Lustre: lustre-MDT0000: Client a7cd2fd9-f632-48f4-8baf-cc2f7ba0bed5 (at 192.168.206.19@tcp) reconnecting [ 3765.741468] Lustre: *** cfs_fail_loc=159, val=0*** [ 3765.743884] Lustre: Skipped 14 previous similar messages [ 3780.892104] Lustre: lustre-MDT0000: Client a7cd2fd9-f632-48f4-8baf-cc2f7ba0bed5 (at 192.168.206.19@tcp) reconnecting [ 3782.934923] Lustre: 139294:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3788.773831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3788.786951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.790415] LustreError: Skipped 1 previous similar message [ 3788.824306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3789.805448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3789.807272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.810800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3794.689465] Lustre: server umount lustre-OST0000 complete [ 3799.009908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3799.023777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.048193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.055340] Lustre: Skipped 1 previous similar message [ 3804.580248] Lustre: server umount lustre-MDT0000 complete [ 3805.156805] LustreError: 135964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3805.182756] LustreError: 135964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3808.624562] LustreError: 138900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836317 with bad export cookie 9444466428605363317 [ 3808.632494] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3808.641331] LustreError: 138900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3809.224362] Lustre: server umount lustre-MDT0001 complete [ 3820.784826] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 3823.794318] Key type lgssc unregistered [ 3824.059628] LNet: 140347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3824.069956] LNetError: 140347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3824.095693] LNet: Removed LNI 192.168.206.119@tcp [ 3824.988924] Key type .llcrypt unregistered [ 3824.991545] Key type ._llcrypt unregistered [ 3848.595794] Key type ._llcrypt registered [ 3848.598630] Key type .llcrypt registered [ 3848.733120] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 3849.724253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3849.755025] alg: No test for adler32 (adler32-zlib) [ 3850.827160] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 3851.145791] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 3852.881428] Key type lgssc registered [ 3854.302880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3861.665038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3861.705885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.295208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3867.589265] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3872.746379] LustreError: 141762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3872.774602] LustreError: 141762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3875.044382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3875.424619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3879.739278] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3886.177150] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3891.585212] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3898.229417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3898.570777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.636485] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3904.492425] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 3907.786285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3910.959786] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3916.097875] Lustre: 144344:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3918.970764] Lustre: 144393:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3920.263118] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3921.069671] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3924.839507] Lustre: 144540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3925.749282] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3926.506175] Lustre: 144638:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3931.620962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3931.627862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.637658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.665536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.672846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.679980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.698769] Lustre: Skipped 1 previous similar message [ 3936.181460] Lustre: server umount lustre-OST0000 complete [ 3941.859679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3941.865772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.881187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.906621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.911387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.929131] Lustre: Skipped 1 previous similar message [ 3949.027762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3949.035244] Lustre: Skipped 1 previous similar message [ 3954.148481] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3954.411833] Lustre: server umount lustre-MDT0000 complete [ 3958.152371] LustreError: 144442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836466 with bad export cookie 10337101094149006255 [ 3958.155499] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3958.170392] LustreError: 144442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3958.611565] Lustre: server umount lustre-MDT0001 complete [ 3968.769722] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 3971.364990] Key type lgssc unregistered [ 3971.665319] LNet: 145710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3971.685062] LNetError: 145710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3971.711587] LNet: Removed LNI 192.168.206.119@tcp [ 3972.559475] Key type .llcrypt unregistered [ 3972.561124] Key type ._llcrypt unregistered [ 3995.883502] Key type ._llcrypt registered [ 3995.886877] Key type .llcrypt registered [ 3996.004882] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 3997.324090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3997.478159] alg: No test for adler32 (adler32-zlib) [ 3998.441883] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 3998.594191] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 4000.232147] Key type lgssc registered [ 4001.728489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4009.868710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4009.917781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.564584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.225371] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4021.737545] LustreError: 147127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4021.745478] LustreError: 147127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4022.395849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.725791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4026.071938] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4032.113354] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4037.209373] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4043.314750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4043.649205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4047.804806] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4049.505574] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4056.522477] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4057.018577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4062.214357] Lustre: *** cfs_fail_loc=159, val=0*** [ 4077.837733] Lustre: lustre-MDT0000: Client 9128ad4c-a4d0-466e-a53e-c22349f826a4 (at 192.168.206.19@tcp) reconnecting [ 4095.976406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.979497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4095.985593] Lustre: Skipped 1 previous similar message [ 4101.104700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4101.113644] Lustre: Skipped 2 previous similar messages [ 4101.334899] Lustre: server umount lustre-OST0000 complete [ 4104.673340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4104.677269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4104.690040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4106.210872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.829466] Lustre: server umount lustre-MDT0000 complete [ 4111.336277] LustreError: 147122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.356980] LustreError: 147122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4114.101372] LustreError: 147109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836622 with bad export cookie 13493449322052264920 [ 4114.104408] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4114.107478] LustreError: 147109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4114.460687] Lustre: server umount lustre-MDT0001 complete [ 4123.973755] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 4126.297882] Key type lgssc unregistered [ 4126.536612] LNet: 150788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4126.554048] LNetError: 150788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4126.578883] LNet: Removed LNI 192.168.206.119@tcp [ 4127.333246] Key type .llcrypt unregistered [ 4127.337675] Key type ._llcrypt unregistered [ 4148.747855] Key type ._llcrypt registered [ 4148.750174] Key type .llcrypt registered [ 4148.886717] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 4149.899046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4149.914979] alg: No test for adler32 (adler32-zlib) [ 4150.956918] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 4151.273122] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 4152.944181] Key type lgssc registered [ 4154.073463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4161.372639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4161.396367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4162.841801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4167.363733] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4173.283752] LustreError: 152204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.305046] LustreError: 152204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4174.612546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.950466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4178.547532] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4185.029730] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4190.264233] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4196.081883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4196.428826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4200.578034] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4202.040640] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4207.728754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4208.879454] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4227.762863] Lustre: lustre-OST0000: evicting f4ede328-7a77-4a5e-b3f7-5f03902857bd (at 192.168.206.19@tcp) by administrative request [ 4227.768357] Lustre: Skipped 1 previous similar message [ 4257.250749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4257.266492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.281126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4259.299745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4259.301133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4259.304264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.318241] Lustre: Skipped 1 previous similar message [ 4264.424777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4264.429901] Lustre: Skipped 1 previous similar message [ 4269.024279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4269.211330] Lustre: server umount lustre-OST0000 complete [ 4274.658495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.660128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4274.666415] Lustre: Skipped 1 previous similar message [ 4278.743838] Lustre: server umount lustre-MDT0000 complete [ 4279.777758] LustreError: 152204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.804942] LustreError: 152204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4282.441075] LustreError: 152184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836790 with bad export cookie 4506053702806650623 [ 4282.446449] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4282.453252] LustreError: 152184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4282.898216] Lustre: server umount lustre-MDT0001 complete [ 4294.072064] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 4297.228872] Key type lgssc unregistered [ 4297.530753] LNet: 156269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4297.539208] LNetError: 156269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4297.569215] LNet: Removed LNI 192.168.206.119@tcp [ 4298.471065] Key type .llcrypt unregistered [ 4298.476768] Key type ._llcrypt unregistered [ 4366.494749] Key type ._llcrypt registered [ 4366.498039] Key type .llcrypt registered [ 4366.588515] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 4367.477376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4367.594726] alg: No test for adler32 (adler32-zlib) [ 4368.738805] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 4368.915986] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 4370.600454] Key type lgssc registered [ 4371.705595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4380.710879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4380.721630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4382.035435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.189978] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4392.428770] LustreError: 158754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.452815] LustreError: 158754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4393.737049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4393.985405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4397.235214] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4403.154378] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4408.576645] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4414.409956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4414.904806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4418.999923] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4421.320556] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4427.181123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4427.366657] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4437.473657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4437.484848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.491846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4437.508201] Lustre: Skipped 1 previous similar message [ 4441.533589] Lustre: server umount lustre-OST0000 complete [ 4445.153656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4445.161705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4445.176950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4445.189059] Lustre: Skipped 1 previous similar message [ 4447.718975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.719924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4447.738847] Lustre: Skipped 1 previous similar message [ 4451.117086] Lustre: server umount lustre-MDT0000 complete [ 4452.834855] LustreError: 160049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.848060] LustreError: 160049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4454.371286] LustreError: 158734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836962 with bad export cookie 6195334599719523963 [ 4454.374550] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4454.378156] LustreError: 158734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4454.571281] Lustre: server umount lustre-MDT0001 complete [ 4463.495052] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 4465.716670] Key type lgssc unregistered [ 4466.005900] LNet: 162291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4466.017456] LNetError: 162291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4466.040444] LNet: Removed LNI 192.168.206.119@tcp [ 4466.816618] Key type .llcrypt unregistered [ 4466.819154] Key type ._llcrypt unregistered [ 4491.404819] Key type ._llcrypt registered [ 4491.409231] Key type .llcrypt registered [ 4491.501152] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 4492.498458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4492.622866] alg: No test for adler32 (adler32-zlib) [ 4493.707044] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 4493.990604] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 4495.648216] Key type lgssc registered [ 4496.459557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4509.888937] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 23:10:15 (1788837015) [ 4524.886731] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 4534.314664] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 4567.809125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4567.826642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.193644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4569.266380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4569.361224] Lustre: lustre-MDT0000: new disk, initializing [ 4569.433862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.449251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4573.737706] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4584.735443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4584.836277] Lustre: 166551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4584.900236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4584.905886] Lustre: Skipped 1 previous similar message [ 4584.941294] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4584.997307] Lustre: lustre-MDT0001: new disk, initializing [ 4585.084931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4585.119580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4585.136840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4589.237983] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4597.598718] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4601.661131] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4606.835428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4607.025929] Lustre: lustre-OST0000: new disk, initializing [ 4607.029318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4607.034318] Lustre: 168516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4607.091293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4611.911460] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4615.276418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4615.294452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4615.402304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4619.736139] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4631.009619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4631.013418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4631.018102] LustreError: Skipped 1 previous similar message [ 4631.035337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4631.047077] Lustre: Skipped 1 previous similar message [ 4633.269846] Lustre: server umount lustre-OST0000 complete [ 4636.132177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4636.132454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.140123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4636.150781] Lustre: Skipped 2 previous similar messages [ 4641.249412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4641.253965] Lustre: Skipped 2 previous similar messages [ 4642.071234] Lustre: server umount lustre-MDT0000 complete [ 4645.249468] LustreError: 167344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837153 with bad export cookie 10666978202885704551 [ 4645.260495] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4645.269323] LustreError: 167344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4645.564531] Lustre: server umount lustre-MDT0001 complete [ 4655.252608] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 4658.108970] Key type lgssc unregistered [ 4658.375855] LNet: 170420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4658.387941] LNetError: 170420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4658.401941] LNet: Removed LNI 192.168.206.119@tcp [ 4659.271568] Key type .llcrypt unregistered [ 4659.276805] Key type ._llcrypt unregistered [ 4681.257972] Key type ._llcrypt registered [ 4681.260622] Key type .llcrypt registered [ 4681.388966] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 4682.446904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4682.522040] alg: No test for adler32 (adler32-zlib) [ 4683.786426] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 4684.057783] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 4685.784157] Key type lgssc registered [ 4687.024308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4694.783650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4694.811458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.236381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4700.691596] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4705.761949] LustreError: 171835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4705.799468] LustreError: 171835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4708.696359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4709.035506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4713.291293] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4720.137640] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4724.956787] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4731.092353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.462061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4737.087077] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 4740.675390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4744.080237] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4750.662072] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4750.683586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4750.696371] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4750.705409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4750.723374] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4751.657501] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4751.681485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.700066] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4751.707196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4751.724235] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4752.698480] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4752.734308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4752.746928] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4752.751835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4752.760527] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4753.588376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4754.456399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4755.309870] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4755.315324] Lustre: Skipped 2 previous similar messages [ 4755.329354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4755.337555] Lustre: Skipped 2 previous similar messages [ 4755.341731] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4755.347646] Lustre: Skipped 2 previous similar messages [ 4755.351093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4755.363781] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4755.376914] Lustre: Skipped 2 previous similar messages [ 4756.257958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4757.070317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4757.876514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4758.705127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.912675] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4759.928767] Lustre: Skipped 4 previous similar messages [ 4759.946309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4759.957257] Lustre: Skipped 4 previous similar messages [ 4759.960914] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4759.966374] Lustre: Skipped 4 previous similar messages [ 4759.970401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.979248] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4759.994280] Lustre: Skipped 4 previous similar messages [ 4760.742606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4761.748456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4762.551641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4763.324586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4764.124684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4764.830957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4765.462663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.079300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.735630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4767.513129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4768.354612] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4768.362129] Lustre: Skipped 10 previous similar messages [ 4768.399212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4768.419463] Lustre: Skipped 10 previous similar messages [ 4768.428354] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4768.437295] Lustre: Skipped 10 previous similar messages [ 4768.444944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4768.465593] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4768.477849] Lustre: Skipped 10 previous similar messages [ 4769.273876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.953770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.692327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.362043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4772.298352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4772.883069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.702905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4774.659502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.481809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.350491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4777.224280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.620055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.005648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4781.473551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.683708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4784.021225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.263866] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4785.285108] Lustre: Skipped 16 previous similar messages [ 4785.308093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.325617] Lustre: Skipped 16 previous similar messages [ 4785.349402] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4785.366608] Lustre: Skipped 16 previous similar messages [ 4785.383173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.399075] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4785.414885] Lustre: Skipped 16 previous similar messages [ 4786.503081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.065614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.891870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.954179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4791.163719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4792.134402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.343536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.546465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.525520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.220844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.206700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.194147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.012861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.725152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.672379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.618655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4802.424234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.466789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.164634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4805.334069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.274224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.026269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4808.083483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4808.772311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.931484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.026690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.937259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4813.340395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4814.137934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4814.990233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.658896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.666329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.759739] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4817.763514] Lustre: Skipped 32 previous similar messages [ 4817.774483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.793569] Lustre: Skipped 32 previous similar messages [ 4817.803308] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4817.812657] Lustre: Skipped 32 previous similar messages [ 4817.830096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.845121] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4817.856463] Lustre: Skipped 32 previous similar messages [ 4818.875574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.821598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4820.728942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4821.635141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.475473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.143559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.967565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4824.698193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4825.576875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.224539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.945970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.844940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.570160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4829.312976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.210447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.924819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4831.702483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.479389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.349111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4834.188047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4835.095793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4835.794781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4836.600481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.365215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.149953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.992871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4839.903885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4840.693844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4841.507477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4842.295534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.162735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.946730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4844.861537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.752915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4846.565682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4847.453083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.388937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4849.254328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.027334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.715446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4851.558187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4852.457638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.319875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4854.234260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.103705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4856.119816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4857.045370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4858.173565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4859.079757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4859.947348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.692966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.486706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4862.343309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4863.371538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4864.312053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4865.177852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4866.160580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.164559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4868.064932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4869.038113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4869.840469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4870.769801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4871.718665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4872.512741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4873.396975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.303631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4875.174161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4876.278327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4877.195455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4878.402310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4879.337643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.457914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4881.329949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.269404] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4882.277656] Lustre: Skipped 73 previous similar messages [ 4882.296323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.309613] Lustre: Skipped 73 previous similar messages [ 4882.321556] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4882.338499] Lustre: Skipped 73 previous similar messages [ 4882.347037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.356995] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4882.378553] Lustre: Skipped 73 previous similar messages [ 4883.204871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.178263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.029655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4886.092325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.035504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.930802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4888.816614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4889.844712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.823814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4891.703983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4892.675432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.656130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4894.816347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.784681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.849486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4897.741977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.700188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4899.599107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4900.546975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4901.440045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4902.397549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4903.192384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4904.050612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4905.031407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4905.891447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4906.700706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4907.585694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4908.441684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4909.110038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4910.021452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4910.833988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4911.676822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4912.512479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4913.407276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4914.242323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4915.117390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4915.932935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4916.694380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4917.485916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.250656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.993243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4919.741640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4920.626101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4921.408515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4922.171427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.077052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.941834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4924.810354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4925.757113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.691509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.508677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4928.338240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4929.072605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.048349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.959136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.827673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4932.722524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4933.768619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4934.600819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4935.542055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4936.604319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4937.564339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4938.358260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.436683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4940.427090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.489860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.397920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.288261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.299379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.293481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4946.414919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.422070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.252538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.450656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.545556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.611385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.661904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4953.443325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.459330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.179729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.003543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.867834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.995898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4958.831093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.796855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.653427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4961.536201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.214690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.916927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.827616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.871936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.781803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.855214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.819042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.024279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.992600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.127096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.438481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.588809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.677490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.660969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.924847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.884472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.255591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.506519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.671579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.535335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.545327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.236854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.149429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.909645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.623310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.342102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.200515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.090155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.811679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.485295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.310262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.951519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.861145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.700726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.588573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.337241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.253444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.058576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.028699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.948025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.842278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.737947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.493615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.631056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.608086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.734458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.969978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.025576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.021625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.906036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.926811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.994879] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5011.000252] Lustre: Skipped 138 previous similar messages [ 5011.014740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5011.024253] Lustre: Skipped 138 previous similar messages [ 5011.029891] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5011.034142] Lustre: Skipped 138 previous similar messages [ 5011.037604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.046314] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5011.050140] Lustre: Skipped 138 previous similar messages [ 5011.967327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.028676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.759555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.897855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.839726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.021812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.008459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.975786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.841505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.774738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5021.762210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.865545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.806669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.714395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.527632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5026.660044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.792457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.645302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.586867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.446853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.393658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.275353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.251169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.188635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.070535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.966677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.997387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.815787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.692515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.688405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.634527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.502795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.409741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.170944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.076058] LustreError: 171854:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5044.089801] LustreError: 171854:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5044.097432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.922530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.768813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.638199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.501647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.259795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.043603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.766808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.553949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.331364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.105693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.974719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.801819] LustreError: 171854:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5053.816916] LustreError: 171854:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5053.821773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.632954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.520952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.383589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.251681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.992526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.858122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.689844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.529773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.265016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.990360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.690531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.444292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.214135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.013209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.788677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.545193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.329909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.104769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.026410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.937261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.754259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.704315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.667307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.587218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.442597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.185835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.036386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.851802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.736595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.507892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.252814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.090638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.888324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.698414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.452416] LustreError: 171854:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5082.464638] LustreError: 171854:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5082.476137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.390574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.356140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.090449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.970658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.714221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.519757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.204498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.019058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.740123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.519795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.371708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.206965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.036435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.859716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.719250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.529800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.264061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.028371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.861697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.595230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.293356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.979886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.645215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.498137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.277540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.085156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.855391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.617603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.390941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.183198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.887911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.711169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.541194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.472802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.182043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.986753] LustreError: 171854:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5110.993320] LustreError: 171854:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5111.003844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.770810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.562314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.293735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.181402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.920169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.740665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.515433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.376901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.094573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.979273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.805822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.584665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.378433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.195930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.926652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.732508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.596593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.594719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.447430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.304335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.043533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.968184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.792368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.718419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.529570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.268793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.017502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.922590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.658127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.646825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.630632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.543280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.446179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.385638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.276396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.070971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.008751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.813938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.633615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.430662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.246137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.085470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.995580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.840791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.629807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.325436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.142816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.055920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.958470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.856401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.763055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.603747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.401024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.197530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.983546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.796962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.713592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.491161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.330899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.027064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.930472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.645553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.474187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.251646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.183401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.282427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.283699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.344217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.411303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.283657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.261197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.520798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.379635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.089485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.953461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.670670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.457418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.357303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.294878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.192795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.343870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.311116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.184866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.049217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.945940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.822364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.737081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.664165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.537344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.493233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.383042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.167288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.074077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.907191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.870038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.618284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.420700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.316561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.100809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.954636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.723783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.544146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.548136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.453846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.231119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.084673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.989711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.836076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.713891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.828621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.706579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.837435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.732927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.671244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.598888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.461566] LustreError: 171854:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -108 [ 5211.482067] LustreError: 171854:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -108 [ 5211.495538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.241191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.258385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.105242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.978670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.916535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.898834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.865944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.714930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.798144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.752838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.730699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.566022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.551679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.314190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.390079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.339995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.212716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.067370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.931669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.864532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.773158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.628561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.631891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.595616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.588496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.536134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.565359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.549425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.460712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.379418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.341561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.186164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.096521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.004625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.906567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.774825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.637526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.489614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.548491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.439258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.217067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.077961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.033142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.879059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.863661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.904164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.985827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.941309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.734396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.838310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.777065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.738795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.683291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.608401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.723956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.582434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.615302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.310340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.196923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.122504] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5267.129889] Lustre: Skipped 293 previous similar messages [ 5267.146753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.159436] Lustre: Skipped 293 previous similar messages [ 5267.166683] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5267.170716] Lustre: Skipped 293 previous similar messages [ 5267.173992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.185205] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5267.192927] Lustre: Skipped 293 previous similar messages [ 5268.136178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.124718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.183708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.338439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.238178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.371664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.480735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.407628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.451912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.275996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.223110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.301796] LustreError: 171854:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5279.319498] LustreError: 171854:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5279.324289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.307078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.350084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.325879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.339403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.396910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.388871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.392553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.416547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.359534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.366182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.336704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.274338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.234938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.111638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.059606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.837033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.817109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.786620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.622304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.443397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.343056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.283838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.265855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.177710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.922890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.909359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.871615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.864733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.739551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.679815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.668989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.466761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.538596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.478546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.358458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.181107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.036767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.911967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.736532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.639289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.503554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.341235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.288734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.370525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.299299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.146949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.983581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.791524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.818355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.695455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.812828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.665825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.546824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.399424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.349310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.220227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.112915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.143799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.080995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.043391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.842973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.719750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.546591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.498264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.475487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.481871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.392883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.342492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.310296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.320336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.255750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.201601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.201257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.232890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.011636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.935451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.991553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.001978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.027322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.908314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.932197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.940675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.850477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.748186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.490335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.390755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.192858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.985936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.906440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.865858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.773408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.480619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.472405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.403422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.417580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.208767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.212031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.181448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.073730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.027504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.970174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.975637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.816057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.662519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.613495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.531622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.423992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.458059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.356945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.399583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.376413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.192486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.046051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.008403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.165289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.152077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.139588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.948711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.818165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.699598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.673796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.624873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.864971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.779503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.763892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.684810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.635563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.625151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.641586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.753734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.619617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.691721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.718721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.536603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.454938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.356041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.479056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.385750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.307519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.123534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.101779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.100082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.002105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.008057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.965906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.838807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.658058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.578405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.283470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.211592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.062671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.876077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.721584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.698872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.530881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.228821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.116344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.958415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.819516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.842928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.781564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.924380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.833307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.785460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.641171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.601838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.490581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.470378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.311886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.358996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.114924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.057179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.099166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.060070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.958180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.933413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.960988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.954486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.920416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.737794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.671967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.497720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.425197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.431693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.383498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.395642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.346755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.370303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.406173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.250324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.232110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.071253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.928362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.897050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.834416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.762490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.579600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.484482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.519758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.535113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.516853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.514776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.501868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.539586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.478476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.380522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.343423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.384340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.287497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.264671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.142969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.034295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.938321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.948661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.852877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.722709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.636507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.570347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.527022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.619354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.596096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.611041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.489039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.359924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.077060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.957888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.921876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.793557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.901631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.884818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.883423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.016443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.902154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.963403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.884128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.943100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.915914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.962235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.856643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.843051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.758750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.785372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.731710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.595964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.472869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.491121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.511694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.568300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.551971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.326722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.217965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.883373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.905927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.715763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.673105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.576462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.555239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.444849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.464183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.480459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.555598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.489749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.617755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.756774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.825337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.977161] LustreError: 171854:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5528.991424] LustreError: 171854:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5529.005865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.028831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.113770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.787945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.667372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.703183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.637739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.551952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.609476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.452144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.400260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.417304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.384744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.297582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.182881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.106977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.054556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.839224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.601030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.436548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.319701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.097395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.936240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.951341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.028267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.012285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.977057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.863119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.710584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.653976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.532588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.465380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.328155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.343663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.267696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.155235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.997137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.991853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.889670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.881077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.733663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.581381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.392693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.187599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.018153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.956612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.773577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.635904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.429878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.229351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.189180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.970826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.865360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.749149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.830673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.838942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.920312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.724634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.593834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.399143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.396726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.260105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.232981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.063306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.951294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.796061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.799810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.508077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.287854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.920586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.807934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.696539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.541579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.363651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.226275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.322986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.054700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.941732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.847641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.878858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.679238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.745188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.704612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.560374] LustreError: 171854:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5603.575683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.575814] LustreError: 171854:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5604.540576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.619959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.463472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.295845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.094864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.969703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.707513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.518509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.247323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.027962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.863636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.619625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.343865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.070923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.877200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.733397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.568728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.407402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.353166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.133255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.971162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.785200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.718069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.600484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.418218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.304073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.198375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.984245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.851851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.638507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.626322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.399223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.360218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.166610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.926406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.623691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.403661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.175591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.960299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.644867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.454421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.258364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.117880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.024098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.975577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.762923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.515572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.364567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.166490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.986725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.890751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.825454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.573176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.452659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.331081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.184345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.020781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.760141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.548524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.244217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.244005] LustreError: 173653:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 5939.259099] LustreError: 171854:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6352.361902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6352.374087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.390188] Lustre: Skipped 421 previous similar messages [ 6352.398101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6353.900823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6353.911503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6353.913354] LustreError: Skipped 1 previous similar message [ 6354.702587] Lustre: server umount lustre-OST0000 complete [ 6360.548909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6360.564281] Lustre: Skipped 1 previous similar message [ 6364.755632] Lustre: server umount lustre-MDT0000 complete [ 6365.671340] LustreError: 225682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.689250] LustreError: 225682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6367.935036] LustreError: 171814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838876 with bad export cookie 16611618535266716589 [ 6367.943956] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6367.952191] LustreError: 171814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6368.293410] Lustre: server umount lustre-MDT0001 complete [ 6378.272603] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 6380.985715] Key type lgssc unregistered [ 6381.261491] LNet: 227022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6381.266399] LNetError: 227022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6381.282478] LNet: Removed LNI 192.168.206.119@tcp [ 6382.129744] Key type .llcrypt unregistered [ 6382.133283] Key type ._llcrypt unregistered [ 6405.785930] Key type ._llcrypt registered [ 6405.796994] Key type .llcrypt registered [ 6405.945805] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 6407.032615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6407.082056] alg: No test for adler32 (adler32-zlib) [ 6408.348599] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 6408.676265] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 6410.537924] Key type lgssc registered [ 6411.841669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6419.799635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6419.824318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6421.381784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6425.931370] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6430.689144] LustreError: 228435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.714475] LustreError: 228435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6434.298913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.706807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6439.163152] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6446.169099] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6450.873654] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6456.925410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6457.268423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6462.664723] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6466.505295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6469.812309] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6474.233586] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6475.614500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.628894] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6475.634928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.648525] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6475.650670] LustreError: 228454:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6480.874473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6480.880803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.895947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6481.667065] Lustre: lustre-OST0000: Not available for connect from 192.168.206.19@tcp (stopping) [ 6481.893414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6483.367279] Lustre: server umount lustre-OST0000 complete [ 6495.672096] LustreError: 228454:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6516.143716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6516.444962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6521.608977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.619635] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6522.147362] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6529.067329] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6530.485211] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6531.892420] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.910694] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6535.510307] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9073981ad800.ost_server_uuid 50 [ 6536.877676] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9073981ad800.ost_server_uuid in FULL state after 0 sec [ 6542.307262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6542.309663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6542.324344] LustreError: Skipped 1 previous similar message [ 6542.325398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6542.337968] Lustre: Skipped 1 previous similar message [ 6542.360906] Lustre: Skipped 3 previous similar messages [ 6547.082377] Lustre: server umount lustre-OST0000 complete [ 6552.545128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6552.554318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.565366] Lustre: Skipped 1 previous similar message [ 6552.569495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6552.578672] Lustre: Skipped 1 previous similar message [ 6556.571940] Lustre: server umount lustre-MDT0000 complete [ 6557.668338] LustreError: 228435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6557.678704] LustreError: 228435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6559.933917] LustreError: 228415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839068 with bad export cookie 11149486156312439567 [ 6559.939312] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6559.957506] LustreError: 228415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6560.476372] Lustre: server umount lustre-MDT0001 complete [ 6572.062210] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 6574.498850] Key type lgssc unregistered [ 6574.792602] LNet: 233638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6574.802044] LNetError: 233638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6574.823265] LNet: Removed LNI 192.168.206.119@tcp [ 6575.585378] Key type .llcrypt unregistered [ 6575.593166] Key type ._llcrypt unregistered [ 6604.697681] Key type ._llcrypt registered [ 6604.702687] Key type .llcrypt registered [ 6604.799376] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 6606.171193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6606.201859] alg: No test for adler32 (adler32-zlib) [ 6607.287784] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 6607.566862] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 6609.336167] Key type lgssc registered [ 6610.550754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6619.147922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6619.174704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.718477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6625.218848] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6630.890175] LustreError: 235363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6630.922754] LustreError: 235363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6633.754869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.254089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6638.812800] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6641.738797] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6648.322265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6648.685318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6654.962545] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6663.587337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6663.815274] Lustre: lustre-OST0001: new disk, initializing [ 6663.820532] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6663.830912] Lustre: 237672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6664.015112] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6666.139103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 6670.459164] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6670.915851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6670.920427] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6671.059489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6682.016300] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6690.286543] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6706.147025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6706.153396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.178792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6707.185858] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.207463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6707.210383] Lustre: Skipped 1 previous similar message [ 6707.231853] Lustre: Skipped 3 previous similar messages [ 6711.589576] Lustre: server umount lustre-MDT0000 complete [ 6712.290041] LustreError: 235363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6712.315334] LustreError: 235363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6715.224229] LustreError: 236871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839223 with bad export cookie 2625817712616099739 [ 6715.234827] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6715.237699] LustreError: 236871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6715.623649] Lustre: server umount lustre-MDT0001 complete [ 6730.088212] Lustre: server umount lustre-OST0000 complete [ 6732.769372] Lustre: 234624:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839225/real 1788839225] req@ffff9f3921427b80 x1875733765017856/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788839241 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6732.826299] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6732.846293] Lustre: Skipped 1 previous similar message [ 6734.465531] Lustre: server umount lustre-OST0001 complete [ 6755.130920] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 6767.023379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.574879] LustreError: 240786:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6767.666992] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.527756] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6772.722776] LustreError: 240787:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.840846] LustreError: 240786:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6781.959254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.299051] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6787.219489] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6790.105408] Lustre: 241927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6797.114520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6803.911835] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6806.704973] LustreError: 242282:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6806.722498] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 6812.264618] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6812.446854] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6812.457695] Lustre: Skipped 1 previous similar message [ 6813.495627] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 6813.635688] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 6818.209606] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6825.593197] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6828.926518] Lustre: 243800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6839.270766] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6839.287469] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6839.292545] Lustre: Skipped 3 previous similar messages [ 6839.305547] Lustre: Skipped 3 previous similar messages [ 6844.389854] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6844.397283] Lustre: Skipped 2 previous similar messages [ 6845.308277] Lustre: server umount mylustre-MDT0000 complete [ 6848.855874] LustreError: 240761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839357 with bad export cookie 2625817712616102609 [ 6848.873042] LustreError: 240761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6848.874793] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6849.130683] Lustre: server umount mylustre-MDT0001 complete [ 6862.957419] Lustre: server umount mylustre-OST0000 complete [ 6864.672592] Lustre: 234624:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839357/real 1788839357] req@ffff9f3924392a00 x1875733765076480/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788839373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6864.702414] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6866.460933] Lustre: server umount mylustre-OST0001 complete [ 6885.613106] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 6894.204910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6894.555251] LustreError: 246105:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6894.573362] LustreError: 246105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6894.633570] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.036915] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6906.375148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.167617] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6914.252669] Lustre: 247246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6920.867800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6927.373781] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6932.460699] LustreError: 247600:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.480214] LustreError: 247600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6932.493279] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 6935.536038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6939.846501] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 6939.854841] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 6941.682314] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 6949.420394] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6953.077976] Lustre: 249116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6960.624494] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.635755] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 6960.645034] Lustre: Skipped 3 previous similar messages [ 6960.670462] Lustre: Skipped 3 previous similar messages [ 6966.287895] Lustre: server umount tfs-MDT0000 complete [ 6969.772752] LustreError: 249119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839478 with bad export cookie 2625817712616104786 [ 6969.776458] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.791217] LustreError: 249119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6970.056908] Lustre: server umount tfs-MDT0001 complete [ 6983.858053] Lustre: server umount tfs-OST0000 complete [ 6987.232156] Lustre: 234626:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839479/real 1788839479] req@ffff9f3927b61f80 x1875733765124992/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788839495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6987.282320] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.010152] Lustre: server umount tfs-OST0001 complete [ 7007.057882] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7015.016759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.331124] LustreError: 251369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7015.381862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7015.392409] Lustre: Skipped 3 previous similar messages [ 7018.840817] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7026.060459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.598319] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7031.749606] Lustre: 252511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7037.308039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7039.806807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7044.014638] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7051.310299] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 7052.934384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7058.461146] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 7060.264344] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7067.605257] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7071.021334] Lustre: 254382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7076.862559] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 23:53:02 (1788839582) [ 7092.194305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.204447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7092.205078] Lustre: Skipped 3 previous similar messages [ 7092.213420] Lustre: Skipped 7 previous similar messages [ 7098.408781] Lustre: server umount lustre-MDT0000 complete [ 7101.821246] LustreError: 254385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839610 with bad export cookie 2625817712616106704 [ 7101.826858] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7101.830612] LustreError: 254385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7111.585856] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7111.589831] Lustre: Skipped 9 previous similar messages [ 7116.256127] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7116.426903] Lustre: server umount lustre-MDT0001 complete [ 7125.295934] Lustre: server umount lustre-OST0000 complete [ 7134.703959] Lustre: server umount lustre-OST0001 complete [ 7140.374106] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 7149.355767] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7180.587740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7180.827617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7180.850832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7180.927383] Lustre: lustre-MDT0000: new disk, initializing [ 7181.003271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7181.006977] Lustre: Skipped 3 previous similar messages [ 7181.048928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7185.828920] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7196.681667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.847467] Lustre: 257573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7196.933920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7196.937987] Lustre: Skipped 1 previous similar message [ 7197.019821] Lustre: lustre-MDT0001: new disk, initializing [ 7197.161193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7197.186086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7201.865226] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7212.047099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7212.410927] Lustre: lustre-OST0000: new disk, initializing [ 7212.421787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7212.432732] Lustre: 259158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7214.022228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7214.042198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7214.191383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7218.917714] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7229.043245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7229.331162] Lustre: lustre-OST0001: new disk, initializing [ 7229.335246] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7229.342447] Lustre: 260030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7230.842007] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7230.855555] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7230.953769] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7235.367508] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7246.554162] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 23:55:52 (1788839752) [ 7259.108522] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 23:56:05 (1788839765) [ 7275.490130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7275.490980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7275.497511] LustreError: Skipped 1 previous similar message [ 7275.499573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7275.507037] Lustre: Skipped 3 previous similar messages [ 7275.518589] Lustre: Skipped 1 previous similar message [ 7287.269311] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7287.396339] Lustre: server umount lustre-OST0000 complete [ 7292.394132] LustreError: 259256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.400946] LustreError: 259256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7296.479450] Lustre: server umount lustre-MDT0000 complete [ 7299.251977] LustreError: 260029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839807 with bad export cookie 2625817712616108482 [ 7299.253780] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7299.261462] LustreError: 260029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7305.614696] Lustre: server umount lustre-MDT0001 complete [ 7314.440174] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 7393.277782] Lustre: server umount lustre-OST0001 complete [ 7398.916702] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 7410.774508] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7438.187178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7438.350975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7438.383264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7438.462206] Lustre: lustre-MDT0000: new disk, initializing [ 7438.521375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.527688] Lustre: Skipped 3 previous similar messages [ 7438.541871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7441.681955] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7449.004220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.121471] Lustre: 264864:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7449.400954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7452.530508] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7460.931436] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7465.553483] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7470.970653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7471.221747] Lustre: lustre-OST0000: new disk, initializing [ 7471.224058] Lustre: Skipped 1 previous similar message [ 7471.228795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7471.239433] Lustre: Skipped 2 previous similar messages [ 7471.243925] Lustre: 266828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7473.200179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7473.204187] Lustre: Skipped 1 previous similar message [ 7473.206239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7473.261062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7476.325220] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7483.986542] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7493.604842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7493.608133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7493.609896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7493.609904] Lustre: Skipped 11 previous similar messages [ 7493.620696] Lustre: Skipped 5 previous similar messages [ 7493.658195] LustreError: Skipped 1 previous similar message [ 7495.090053] Lustre: server umount lustre-OST0000 complete [ 7504.642381] Lustre: server umount lustre-MDT0000 complete [ 7507.878531] LustreError: 264856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840016 with bad export cookie 2625817712616109952 [ 7507.885101] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7507.899856] LustreError: 264856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7508.236280] Lustre: server umount lustre-MDT0001 complete [ 7517.437675] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 7519.743873] Key type lgssc unregistered [ 7520.036801] LNet: 268590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7520.047809] LNetError: 268590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7520.062447] LNet: Removed LNI 192.168.206.119@tcp [ 7520.750162] Key type .llcrypt unregistered [ 7520.754373] Key type ._llcrypt unregistered [ 7543.947479] Key type ._llcrypt registered [ 7543.950172] Key type .llcrypt registered [ 7544.042924] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7544.973522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7545.000032] alg: No test for adler32 (adler32-zlib) [ 7546.009272] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 7546.181583] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 7547.849486] Key type lgssc registered [ 7548.894504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7556.803785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7556.828519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7558.156492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7562.557743] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7568.361549] LustreError: 270061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7568.394675] LustreError: 270061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7571.211756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.566163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7575.362879] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7581.765631] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7586.345804] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7591.929358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7592.222260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.756943] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7603.516780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7604.093494] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7604.893139] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 7606.753926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7606.759281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.770117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7608.806735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7608.811266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7608.811309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7608.826334] Lustre: Skipped 1 previous similar message [ 7612.635658] Lustre: server umount lustre-OST0000 complete [ 7617.506883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7617.523962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7617.537462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7624.162642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7624.168809] Lustre: Skipped 3 previous similar messages [ 7629.287603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7629.294568] Lustre: Skipped 1 previous similar message [ 7630.304340] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7630.486671] Lustre: server umount lustre-MDT0000 complete [ 7633.694975] LustreError: 272592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840142 with bad export cookie 10093260081775102475 [ 7633.702577] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7633.709760] LustreError: 272592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7634.025198] Lustre: server umount lustre-MDT0001 complete [ 7646.054503] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 7648.724848] Key type lgssc unregistered [ 7649.134935] LNet: 273648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7649.147401] LNetError: 273648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7649.165635] LNet: Removed LNI 192.168.206.119@tcp [ 7650.063207] Key type .llcrypt unregistered [ 7650.066372] Key type ._llcrypt unregistered [ 7666.391929] Key type ._llcrypt registered [ 7666.397809] Key type .llcrypt registered [ 7666.530357] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7667.347906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7667.499615] alg: No test for adler32 (adler32-zlib) [ 7668.563114] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 7668.774295] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 7670.456173] Key type lgssc registered [ 7671.581352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7679.498152] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7679.537819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7695.074190] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7700.192322] LustreError: 274837:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.119@tcp: failed processing log, type 4: rc = -110 [ 7726.880266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.797576] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7739.330788] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7741.230455] Lustre: server umount lustre-OST0000 complete [ 7755.810588] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 7758.048110] Key type lgssc unregistered [ 7758.290784] LNet: 276470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7758.299693] LNetError: 276470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7759.334810] LNet: Removed LNI 192.168.206.119@tcp [ 7759.996904] Key type .llcrypt unregistered [ 7759.999379] Key type ._llcrypt unregistered [ 7798.100382] Key type ._llcrypt registered [ 7798.107104] Key type .llcrypt registered [ 7798.251210] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7799.160031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7799.276661] alg: No test for adler32 (adler32-zlib) [ 7800.313138] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 7800.521029] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 7802.200130] Key type lgssc registered [ 7803.586367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7816.284504] loop0: detected capacity change from 0 to 204800000 [ 7816.466303] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7816.472905] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7816.740822] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7817.770935] loop0: detected capacity change from 0 to 204800000 [ 7817.989694] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7817.998916] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7818.324554] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7819.411375] loop0: detected capacity change from 0 to 204800000 [ 7819.676599] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7819.686764] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7819.790877] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7820.980784] loop0: detected capacity change from 0 to 204800000 [ 7821.160982] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7822.381133] loop0: detected capacity change from 0 to 204800000 [ 7822.413507] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7826.543674] loop0: detected capacity change from 0 to 204800000 [ 7826.568656] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7830.415267] loop0: detected capacity change from 0 to 204800000 [ 7830.450254] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7834.492833] loop0: detected capacity change from 0 to 204800000 [ 7834.513275] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7838.396228] loop0: detected capacity change from 0 to 204800000 [ 7838.519929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7838.569628] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.672970] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7838.684769] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 7838.703275] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7838.715737] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 7838.726237] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7838.738235] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 7838.754042] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7838.768493] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 7838.780852] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 7838.788850] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 7838.798774] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 7838.807209] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 7838.815978] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 7838.825535] Lustre: 279881:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 7844.450334] Lustre: server umount MGS complete [ 7845.255701] loop0: detected capacity change from 0 to 204800000 [ 7845.400515] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7846.218976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.272445] Lustre: lustre-MDT0000: Aborting client recovery [ 7847.094224] loop1: detected capacity change from 0 to 204800000 [ 7847.292931] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7847.387690] Lustre: 280313:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7847.405614] Lustre: 280313:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7847.420684] Lustre: 280313:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 7847.435019] Lustre: 280313:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7847.453954] Lustre: 280313:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7847.469607] Lustre: 280313:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 7847.485053] Lustre: 280313:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 7847.517619] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7847.706191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7847.769343] Lustre: lustre-MDT0001: Aborting client recovery [ 7848.685580] loop2: detected capacity change from 0 to 204800000 [ 7848.876389] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7848.945314] Lustre: 280393:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7848.964256] Lustre: 280393:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7848.979401] Lustre: 280393:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7848.996394] Lustre: 280393:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7849.003137] Lustre: 280393:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7849.011446] Lustre: 280393:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7849.019246] Lustre: 280393:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7849.145253] Lustre: Found index 0 for lustre-OST0000, updating log [ 7849.193885] Lustre: 280412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7849.296407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7849.318282] Lustre: lustre-OST0000: Aborting recovery [ 7850.419941] loop3: detected capacity change from 0 to 204800000 [ 7850.507432] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7850.585849] Lustre: 280478:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7850.596351] Lustre: 280478:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7850.614806] Lustre: 280478:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7850.627487] Lustre: 280478:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7850.644662] Lustre: 280478:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7850.660546] Lustre: 280478:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7850.671763] Lustre: 280478:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7850.707354] Lustre: Found index 1 for lustre-OST0001, updating log [ 7850.773770] Lustre: 280483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7850.966587] Lustre: lustre-OST0001: Aborting recovery [ 7856.036409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7856.045598] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7856.072625] LustreError: 280257:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 7856.096469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 7856.096541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 7858.095766] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 7858.100077] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 7858.659608] LustreError: 280224:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1360us: rc = -2 [ 7859.160365] LustreError: 280224:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 457us: rc = -2 [ 7859.172970] LustreError: 280224:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 7873.505074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7873.505469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7873.507666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7873.517636] Lustre: Skipped 3 previous similar messages [ 7878.625193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7878.627593] Lustre: Skipped 3 previous similar messages [ 7883.745510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7883.750542] Lustre: Skipped 3 previous similar messages [ 7885.280141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7885.569571] Lustre: server umount lustre-MDT0000 complete [ 7886.488601] LustreError: 280208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840394 with bad export cookie 15825207794809369128 [ 7886.495782] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7886.514071] LustreError: 280208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7888.865093] LustreError: 280223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7888.865378] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.894699] LustreError: 280223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7888.939104] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7888.942046] Lustre: Skipped 1 previous similar message [ 7891.888744] LustreError: 280222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7891.906607] LustreError: 280222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7893.075901] Lustre: server umount lustre-MDT0001 complete [ 7900.301181] Lustre: server umount lustre-OST0000 complete [ 7900.902040] LustreError: 280979:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000bd0:0x2:0x0].0x0 (00000000209fd930) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7907.114591] Lustre: server umount lustre-OST0001 complete [ 7914.972624] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 00:07:00 (1788840420) [ 7940.853405] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 7950.385101] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 7983.972273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7984.140331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7984.166605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7984.251061] Lustre: lustre-MDT0000: new disk, initializing [ 7984.341682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7984.351077] Lustre: Skipped 1 previous similar message [ 7984.371315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7988.671921] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 7999.021421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.164763] Lustre: 284551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7999.213259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7999.216535] Lustre: Skipped 1 previous similar message [ 7999.282748] Lustre: lustre-MDT0001: new disk, initializing [ 7999.356130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7999.412038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7999.425068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8004.590446] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8013.961156] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8019.053826] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8024.530311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8024.775535] Lustre: lustre-OST0000: new disk, initializing [ 8024.781541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8024.791703] Lustre: 286517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8024.853936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8030.578981] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8033.872034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8033.886372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8034.045191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8039.668292] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8044.773917] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8056.373186] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8062.168970] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8067.765549] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8114.158140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.174911] Lustre: Skipped 1 previous similar message [ 8114.190268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.200979] Lustre: Skipped 2 previous similar messages [ 8119.266120] LustreError: 286500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8119.297555] LustreError: 286500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8119.612973] Lustre: server umount lustre-OST0000 complete [ 8124.399450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.420130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8124.425892] Lustre: Skipped 2 previous similar messages [ 8124.446231] Lustre: Skipped 1 previous similar message [ 8130.052478] Lustre: server umount lustre-MDT0000 complete [ 8134.193929] LustreError: 284544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840642 with bad export cookie 15825207794809371837 [ 8134.195944] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8134.201613] LustreError: 284544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8134.407323] Lustre: server umount lustre-MDT0001 complete [ 8144.836817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8150.184407] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8155.367509] Lustre: server umount MGS complete [ 8166.517408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8166.953256] LustreError: 290391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8166.979521] LustreError: 290404:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8167.044804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8171.372482] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8172.514983] LustreError: 290392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8172.537620] LustreError: 290392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8179.094903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8183.838539] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8190.364464] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8195.607522] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8201.974738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8202.254212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8202.262355] Lustre: Skipped 1 previous similar message [ 8207.658931] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8213.451073] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8238.051604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8238.056083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.064844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.068114] Lustre: Skipped 1 previous similar message [ 8238.077216] Lustre: Skipped 2 previous similar messages [ 8242.496774] Lustre: server umount lustre-OST0000 complete [ 8246.242099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8246.247824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.897399] Lustre: server umount lustre-MDT0000 complete [ 8253.411026] LustreError: 291921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.424596] LustreError: 291921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8254.867774] LustreError: 293227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840763 with bad export cookie 15825207794809373685 [ 8254.870629] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.886631] LustreError: 293227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8255.168304] Lustre: server umount lustre-MDT0001 complete [ 8264.663263] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 8267.135743] Key type lgssc unregistered [ 8267.427148] LNet: 294332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8267.449935] LNetError: 294332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8267.491860] LNet: Removed LNI 192.168.206.119@tcp [ 8268.327595] Key type .llcrypt unregistered [ 8268.330085] Key type ._llcrypt unregistered [ 8299.712361] Key type ._llcrypt registered [ 8299.714347] Key type .llcrypt registered [ 8299.819377] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 8312.231479] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 8313.065500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8313.159103] alg: No test for adler32 (adler32-zlib) [ 8314.151133] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 8314.452993] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 8316.136158] Key type lgssc registered [ 8317.469706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8349.979147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8350.014241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8351.300657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8351.333888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8351.437206] Lustre: lustre-MDT0000: new disk, initializing [ 8351.558235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8351.590426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8355.340470] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8364.920035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.039377] Lustre: 298265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8365.094954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8365.098420] Lustre: Skipped 1 previous similar message [ 8365.190000] Lustre: lustre-MDT0001: new disk, initializing [ 8365.300154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8365.346696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8365.363691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8369.397272] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8377.460910] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8382.553353] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8388.449894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8388.818051] Lustre: lustre-OST0000: new disk, initializing [ 8388.823458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8388.830673] Lustre: 300237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8388.930902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8394.115587] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8395.806523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8395.817333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8395.903349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8401.460220] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8407.897148] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8419.554768] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8427.820499] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8433.845789] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8482.796180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.798080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8482.802984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8482.814069] Lustre: Skipped 1 previous similar message [ 8485.545827] Lustre: server umount lustre-OST0000 complete [ 8493.026783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8493.032705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8493.042825] Lustre: Skipped 1 previous similar message [ 8493.055444] Lustre: Skipped 2 previous similar messages [ 8498.145615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8498.154776] Lustre: Skipped 1 previous similar message [ 8503.267477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8503.272244] Lustre: Skipped 1 previous similar message [ 8504.288322] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8504.450647] Lustre: server umount lustre-MDT0000 complete [ 8507.118123] LustreError: 298256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841015 with bad export cookie 4182623742449104613 [ 8507.124190] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8507.130779] LustreError: 298256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8507.334150] Lustre: server umount lustre-MDT0001 complete [ 8513.653281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.141690] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8520.705425] Lustre: server umount MGS complete [ 8529.323958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8529.638584] LustreError: 304266:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8529.691681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8533.010609] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8535.020818] LustreError: 304253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8535.039071] LustreError: 304253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8539.540168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8539.875288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8543.618228] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8549.172614] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8553.288208] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8558.722639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8559.051678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.558369] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8569.310643] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8607.206590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8607.209195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8607.213201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8607.217485] Lustre: Skipped 1 previous similar message [ 8609.093996] Lustre: server umount lustre-OST0000 complete [ 8612.326158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8612.337286] Lustre: Skipped 1 previous similar message [ 8617.443809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8617.451072] Lustre: Skipped 4 previous similar messages [ 8618.286668] Lustre: server umount lustre-MDT0000 complete [ 8621.373417] LustreError: 304235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841129 with bad export cookie 4182623742449106566 [ 8621.375175] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8621.382527] LustreError: 304235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8621.668857] Lustre: server umount lustre-MDT0001 complete [ 8630.721623] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 8633.658810] Key type lgssc unregistered [ 8634.016397] LNet: 308215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8634.021301] LNetError: 308215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8634.060840] LNet: Removed LNI 192.168.206.119@tcp [ 8634.974439] Key type .llcrypt unregistered [ 8634.978577] Key type ._llcrypt unregistered [ 8653.298940] Key type ._llcrypt registered [ 8653.300312] Key type .llcrypt registered [ 8653.411837] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 8665.511674] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 8666.472994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8666.494761] alg: No test for adler32 (adler32-zlib) [ 8667.625399] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 8667.948286] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 8669.729660] Key type lgssc registered [ 8671.068723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8689.739977] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 8701.601791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8701.617273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.798472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8702.818426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8702.883518] Lustre: lustre-MDT0000: new disk, initializing [ 8702.950369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8702.978884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8706.532452] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 8714.901336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8715.102838] Lustre: lustre-OST0000: new disk, initializing [ 8715.108422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8715.111818] Lustre: Skipped 1 previous similar message [ 8715.115480] Lustre: 312242:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8715.166063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8717.131974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8717.141740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8717.171587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8719.498630] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9437.156474] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9437.170446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9442.279613] Lustre: server umount lustre-MDT0000 complete [ 9445.272495] LustreError: 311428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841953 with bad export cookie 2611551789128323599 [ 9445.282809] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9445.283771] LustreError: 311428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9445.407299] Lustre: server umount lustre-OST0000 complete [ 9467.563368] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [ 9476.280357] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 9506.076457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9506.233207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9506.256365] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9506.307340] Lustre: lustre-MDT0000: new disk, initializing [ 9506.358407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9506.371889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9511.573932] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9521.896390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9522.005831] Lustre: 316554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9522.049644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9522.053628] Lustre: Skipped 1 previous similar message [ 9522.150150] Lustre: lustre-MDT0001: new disk, initializing [ 9522.245295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9522.304550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9522.323463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9526.362713] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9534.891103] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9539.706371] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9545.540917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9545.886789] Lustre: lustre-OST0000: new disk, initializing [ 9545.890925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9545.897433] Lustre: 318520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9545.978455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9551.404560] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9551.426567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9551.434429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9551.591702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9560.248559] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9564.761649] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9564.980698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9569.267779] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9569.513911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9572.335270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9572.352517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.353641] LustreError: Skipped 1 previous similar message [ 9572.355647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9572.401330] Lustre: Skipped 1 previous similar message [ 9577.447633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9577.455224] Lustre: Skipped 2 previous similar messages [ 9577.812348] Lustre: server umount lustre-OST0000 complete [ 9582.562958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9582.576623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9582.588588] Lustre: Skipped 1 previous similar message [ 9582.611654] Lustre: Skipped 1 previous similar message [ 9587.578859] Lustre: server umount lustre-MDT0000 complete [ 9590.354473] LustreError: 317357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842098 with bad export cookie 2611551789131264418 [ 9590.358780] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9590.365059] LustreError: 317357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9590.650797] Lustre: server umount lustre-MDT0001 complete [ 9593.196537] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 9594.507855] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 00:35:00 (1788842100) [ 9599.870888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9600.186546] LustreError: 320833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9600.207713] LustreError: 320833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9600.249163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9604.419299] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9605.603910] LustreError: 320834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9609.702049] LustreError: 320833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9611.548154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9611.787946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9615.445153] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9621.336391] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9625.877779] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9631.273771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9631.605661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9636.059438] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9641.957640] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9648.605547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9648.787299] Lustre: lustre-OST0001: new disk, initializing [ 9648.790611] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9648.803043] Lustre: 323668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9648.920458] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9654.881690] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9655.482788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9655.498962] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9655.616544] LustreError: 324046:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9655.623360] LustreError: 324046:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 9656.684372] LustreError: 324050:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 9656.696305] LustreError: 324050:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 9657.698584] LustreError: 324046:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9659.751395] LustreError: 324046:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9659.761250] LustreError: 324046:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 9663.916445] LustreError: 324050:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 9663.929588] LustreError: 324050:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [ 9664.465525] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9664.629937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9669.506434] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff907388ac7000.ost_server_uuid 50 [ 9670.973607] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff907388ac7000.ost_server_uuid in FULL state after 0 sec [ 9672.100609] LustreError: 324046:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9672.112375] LustreError: 324046:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [ 9676.414813] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff907388ac7000.ost_server_uuid 50 [ 9677.715214] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff907388ac7000.ost_server_uuid in FULL state after 0 sec [ 9679.431468] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9694.689213] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9694.690285] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.694753] LustreError: Skipped 1 previous similar message [ 9694.708918] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9694.719771] Lustre: Skipped 1 previous similar message [ 9698.474530] Lustre: server umount lustre-OST0001 complete [ 9701.088852] LustreError: 323426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.468926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9706.473304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9706.487516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9706.490487] Lustre: Skipped 4 previous similar messages [ 9707.003486] LustreError: 323426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9707.019059] LustreError: 323426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9709.850483] Lustre: server umount lustre-OST0000 complete [ 9714.152088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9717.226248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9717.244075] Lustre: Skipped 2 previous similar messages [ 9719.029359] Lustre: server umount lustre-MDT0000 complete [ 9722.183176] LustreError: 323667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842230 with bad export cookie 2611551789131265300 [ 9722.184930] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9722.203155] LustreError: 323667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9722.521247] Lustre: server umount lustre-MDT0001 complete [ 9730.867713] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 9733.386855] Key type lgssc unregistered [ 9733.687281] LNet: 326351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9733.691207] LNetError: 326351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9733.705819] LNet: Removed LNI 192.168.206.119@tcp [ 9734.618820] Key type .llcrypt unregistered [ 9734.621703] Key type ._llcrypt unregistered [ 9755.188215] Key type ._llcrypt registered [ 9755.190429] Key type .llcrypt registered [ 9755.284203] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [ 9756.061827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9756.081773] alg: No test for adler32 (adler32-zlib) [ 9757.026155] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [ 9757.199239] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [ 9758.840188] Key type lgssc registered [ 9759.801274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9765.777709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9765.798042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9767.160697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9770.755499] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9776.611606] LustreError: 327762:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9776.622810] LustreError: 327762:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9777.080103] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9781.718392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9781.729618] LustreError: 327763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9782.027071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9784.569374] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9791.208921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9791.539091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9792.549880] LustreError: 329442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.563060] LustreError: 329442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9792.567414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 9796.422235] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9797.611198] LustreError: 329532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.622810] LustreError: 329532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9801.831971] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9802.723704] LustreError: 329442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9802.737698] LustreError: 329442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9806.862086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9807.040223] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9811.595058] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [ 9812.468060] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 9817.113107] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9822.414728] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9822.624977] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9925.089266] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9925.100616] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9925.109991] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9925.113812] Lustre: Skipped 1 previous similar message [ 9930.215897] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9930.226021] Lustre: Skipped 2 previous similar messages [ 9935.330583] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9936.352122] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9936.466705] Lustre: server umount lustre-OST0001 complete [ 9940.455342] LustreError: 329442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9940.472502] LustreError: 329442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9940.961590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9940.965595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.970582] LustreError: Skipped 2 previous similar messages [ 9940.975297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9940.983611] Lustre: Skipped 1 previous similar message [ 9940.998080] Lustre: Skipped 2 previous similar messages [ 9945.572603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9945.585746] Lustre: Skipped 1 previous similar message [ 9955.296148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9955.451909] Lustre: server umount lustre-OST0000 complete [ 9960.930640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9960.934440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9960.945741] Lustre: Skipped 1 previous similar message [ 9960.956950] Lustre: Skipped 3 previous similar messages [ 9966.050106] LustreError: 331443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9966.080643] LustreError: 331443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9966.127712] Lustre: server umount lustre-MDT0000 complete [ 9969.833190] LustreError: 327742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842478 with bad export cookie 5448108893760853558 [ 9969.839482] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9969.841592] LustreError: 327742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9970.150494] Lustre: server umount lustre-MDT0001 complete [ 9979.814723] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [ 9982.712334] Key type lgssc unregistered [ 9982.994570] LNet: 332701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9983.001825] LNetError: 332701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9983.026029] LNet: Removed LNI 192.168.206.119@tcp [ 9983.886317] Key type .llcrypt unregistered [ 9983.888412] Key type ._llcrypt unregistered [10026.078344] Key type ._llcrypt registered [10026.084208] Key type .llcrypt registered [10026.250359] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [10027.639476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10027.743669] alg: No test for adler32 (adler32-zlib) [10028.819266] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [10029.097634] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [10030.768320] Key type lgssc registered [10032.381357] Lustre: Echo OBD driver; http://www.lustre.org/ [10042.231602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10042.298398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10043.921820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10049.076782] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10054.122088] LustreError: 334988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10054.162334] LustreError: 334988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10057.605402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10058.008888] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10062.460468] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10065.368493] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10071.823249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10072.238494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10078.271607] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10083.504806] LustreError: 336479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10086.945257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10087.273499] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10089.423848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10089.431948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10093.749825] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10103.673800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10112.134685] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10124.769512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10124.781799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10124.809330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10125.294727] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10125.310288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10125.313504] Lustre: Skipped 2 previous similar messages [10125.330909] Lustre: Skipped 3 previous similar messages [10129.848311] Lustre: server umount lustre-MDT0000 complete [10130.411857] LustreError: 334988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.433938] LustreError: 334988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10133.610235] LustreError: 334967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842642 with bad export cookie 1288229704324200673 [10133.622349] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10133.631735] LustreError: 334967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10133.948265] Lustre: server umount lustre-MDT0001 complete [10147.612917] Lustre: server umount lustre-OST0000 complete [10161.504946] Lustre: server umount lustre-OST0001 complete [10178.833910] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [10187.349456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10187.554380] LustreError: 340254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.621094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10191.242689] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10192.866089] LustreError: 340255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10198.034697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10198.391352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10201.924852] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10203.994478] Lustre: 341395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10210.064299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10212.588301] LustreError: 341747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10212.615645] LustreError: 341747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10212.620778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10218.795588] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10229.018791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10229.345596] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10229.355252] Lustre: Skipped 1 previous similar message [10234.363827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [10235.519292] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10241.610845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10244.653496] Lustre: 343262:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10249.704452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10249.712525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10249.719785] Lustre: Skipped 2 previous similar messages [10249.723809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10249.733888] Lustre: Skipped 3 previous similar messages [10254.118656] Lustre: server umount lustre-MDT0000 complete [10254.832291] LustreError: 340255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10254.857247] LustreError: 340255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10256.815489] LustreError: 340236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842765 with bad export cookie 1288229704324202381 [10256.819851] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10256.821364] LustreError: 340236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10257.117825] Lustre: server umount lustre-MDT0001 complete [10269.874112] Lustre: server umount lustre-OST0000 complete [10283.495289] Lustre: server umount lustre-OST0001 complete [10288.188030] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [10289.767497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [10291.575210] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 00:46:37 (1788842797) [10315.693331] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [10324.119825] loop0: detected capacity change from 0 to 1099511627776 [10324.600516] SGI XFS with ACLs, security attributes, quota, no debug enabled [10324.605480] *** VALIDATE xfs *** [10324.629234] XFS (loop0): Mounting V5 Filesystem [10324.674677] XFS (loop0): Ending clean mount [10325.738959] loop1: detected capacity change from 0 to 18691697672192 [10423.087159] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [10436.192217] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [10468.152158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10468.431699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10468.455065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10468.518157] Lustre: lustre-MDT0000: new disk, initializing [10468.586201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10468.599090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10472.466272] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10480.979523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10481.092739] Lustre: 348394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10481.157825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10481.162930] Lustre: Skipped 1 previous similar message [10481.235366] Lustre: lustre-MDT0001: new disk, initializing [10481.328592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10481.336613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10484.326171] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10493.284760] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10500.110110] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10506.170628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10506.398307] Lustre: lustre-OST0000: new disk, initializing [10506.402256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10506.408323] Lustre: 350365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10507.865265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10507.876359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10508.000190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10511.576124] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10518.619367] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10521.945809] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10522.176270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10526.208588] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10526.425775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10528.749485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10528.751438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10528.760422] Lustre: Skipped 1 previous similar message [10528.773373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10528.779375] Lustre: Skipped 1 previous similar message [10533.889698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10533.911570] Lustre: Skipped 1 previous similar message [10534.522550] Lustre: server umount lustre-OST0000 complete [10538.984090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10539.006028] Lustre: Skipped 2 previous similar messages [10544.098891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10544.105166] Lustre: Skipped 3 previous similar messages [10553.312109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10553.593445] Lustre: server umount lustre-MDT0000 complete [10554.341899] LustreError: 348403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10554.383910] LustreError: 348403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10557.417822] LustreError: 348389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843065 with bad export cookie 1288229704324203928 [10557.428057] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10557.443405] LustreError: 348389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10557.733161] Lustre: server umount lustre-MDT0001 complete [10566.163775] XFS (loop0): Unmounting Filesystem [10570.507988] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 00:51:16 (1788843076) [10576.813652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10577.237842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10577.244023] Lustre: Skipped 2 previous similar messages [10581.236882] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10588.557634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10593.035606] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10599.784421] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10605.230306] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10611.408800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10617.245035] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10623.553345] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10632.161414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10632.165856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10632.173815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10632.191589] Lustre: Skipped 1 previous similar message [10632.201463] Lustre: Skipped 3 previous similar messages [10637.268668] Lustre: server umount lustre-OST0000 complete [10646.929904] Lustre: server umount lustre-MDT0000 complete [10647.525985] LustreError: 352920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.554733] LustreError: 352920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10650.573488] LustreError: 352904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843159 with bad export cookie 1288229704324204810 [10650.596972] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10650.985203] Lustre: server umount lustre-MDT0001 complete [10663.013923] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [10666.612713] Key type lgssc unregistered [10666.991777] LNet: 356549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10667.000675] LNetError: 356549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10667.018166] LNet: Removed LNI 192.168.206.119@tcp [10667.825239] Key type .llcrypt unregistered [10667.832068] Key type ._llcrypt unregistered [10693.037204] Key type ._llcrypt registered [10693.039294] Key type .llcrypt registered [10693.134344] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [10694.234993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10694.412180] alg: No test for adler32 (adler32-zlib) [10695.404473] Lustre: Lustre: Build Version: 2.17.57_105_g5f3dd0c [10695.660290] LNet: Added LNI 192.168.206.119@tcp [8/256/0/180] [10697.321278] Key type lgssc registered [10698.319647] Lustre: Echo OBD driver; http://www.lustre.org/ [10706.529227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10706.552055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10708.012636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10712.076177] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10718.182360] LustreError: 358113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.205029] LustreError: 358113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10719.404809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10719.896621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10723.948369] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10730.965702] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10736.395214] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10742.119035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10742.418754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10747.393852] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10754.732321] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10764.259215] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10764.269640] Lustre: Skipped 1 previous similar message [10764.269798] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10764.280526] Lustre: Skipped 1 previous similar message [10767.604264] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.19@tcp (stopping) [10769.362216] Lustre: server umount lustre-MDT0001 complete [10769.389743] LustreError: 358108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.410626] LustreError: 358108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10772.706829] LustreError: 358109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.825264] LustreError: 358109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10777.843547] LustreError: 358109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10781.047784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.212858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10781.216256] Lustre: Found index 1 for lustre-MDT0001, updating log [10781.245676] Lustre: 358101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10781.507510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10785.464188] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [10786.813599] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10790.385475] LustreError: 361713:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [10790.392183] LustreError: 361713:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [10790.405320] LustreError: 358105:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [10790.442846] LustreError: 361716:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.119@tcp: cfg command failed: rc = -17 [10790.453723] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [10790.453723] [11093.996782] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11094.001463] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11094.016158] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11094.035967] Lustre: Skipped 1 previous similar message [11097.575340] Lustre: server umount lustre-MDT0001 complete [11099.107724] LustreError: 358109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.125087] LustreError: 358109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11107.441027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11107.593459] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11107.596386] Lustre: Found index 1 for lustre-MDT0001, updating log [11107.634078] Lustre: 358102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11108.004275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11112.504865] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11113.449837] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11115.526503] LustreError: 370538:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11115.539558] LustreError: 370538:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11115.552327] LustreError: 358105:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11115.557851] LustreError: 358105:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11115.606927] LustreError: 370541:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11115.612824] LustreError: 370541:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11115.617507] LustreError: 370541:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.119@tcp: cfg command failed: rc = -17 [11115.623923] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.206.119@tcp [11115.623923] [11417.858202] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.19@tcp (stopping) [11417.870838] Lustre: Skipped 2 previous similar messages [11420.642572] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11420.651600] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11420.673381] Lustre: Skipped 1 previous similar message [11422.941228] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.19@tcp (stopping) [11422.949636] Lustre: Skipped 2 previous similar messages [11423.986427] Lustre: server umount lustre-MDT0001 complete [11425.763860] LustreError: 360741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.793423] LustreError: 360741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11433.950593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11434.019064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11434.026240] Lustre: Found index 1 for lustre-MDT0001, updating log [11434.070875] Lustre: 358101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11434.346982] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11437.986983] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11439.601812] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11441.650032] LustreError: 379823:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11441.670366] LustreError: 379823:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11441.677518] LustreError: 358105:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11441.683547] LustreError: 358105:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11441.734575] LustreError: 379826:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [11441.752626] LustreError: 379826:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.119@tcp: cfg command failed: rc = -114 [11441.761480] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [11441.761480] [11762.159631] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11762.166936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11762.193474] Lustre: Skipped 2 previous similar messages [11762.194109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11762.221479] Lustre: Skipped 2 previous similar messages [11765.435666] Lustre: server umount lustre-MDT0000 complete [11767.269065] LustreError: 360741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.295444] LustreError: 360741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11769.797307] LustreError: 358095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844278 with bad export cookie 10492727602933186918 [11769.811238] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11769.821960] LustreError: 358095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11772.386215] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11779.560113] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11779.567425] Lustre: Skipped 2 previous similar messages [11784.160284] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11784.495896] Lustre: server umount lustre-MDT0001 complete [11787.695763] Lustre: server umount lustre-OST0000 complete [11795.348847] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [11805.074578] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [11837.790650] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 01:12:23 (1788844343) [11844.109865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11844.306619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11844.334662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11844.411236] Lustre: lustre-MDT0000: new disk, initializing [11844.475457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11844.490235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11848.195236] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11856.455041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11856.551875] Lustre: 391604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11856.591959] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11856.595200] Lustre: Skipped 1 previous similar message [11856.648505] Lustre: lustre-MDT0001: new disk, initializing [11856.706251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11856.731212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11856.739824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11860.241968] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11868.522571] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11873.899714] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11879.978137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11880.228058] Lustre: lustre-OST0000: new disk, initializing [11880.233563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11880.242651] Lustre: 393571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11880.344387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11885.152147] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_default_debug -1 all [11886.123341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11886.140633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11886.229298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11892.678686] Lustre: DEBUG MARKER: oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11897.825352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11897.832873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11897.846749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11903.733536] Lustre: server umount lustre-MDT0000 complete [11907.046342] LustreError: 391611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11907.070253] LustreError: 391611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11907.112600] LustreError: 391596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844415 with bad export cookie 10492727602933188878 [11907.121154] LustreError: MGC192.168.206.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11907.123079] LustreError: 392410:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [11907.133237] LustreError: 391596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11907.464690] Lustre: server umount lustre-MDT0001 complete [11921.489362] Lustre: server umount lustre-OST0000 complete [11948.133680] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing set_hostid [11958.479982] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing load_modules_local [11987.218917] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11691 sec ========= 01:14:53 (1788844493) [11988.750581] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:14:54 (1788844494) === [11991.778466] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:14:57 (1788844497) === [12028.316733] Lustre: DEBUG MARKER: oleg619-server.virtnet: executing unload_modules_local [12030.415374] Key type lgssc unregistered [12030.646798] LNet: 399932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12030.659126] LNetError: 399932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12030.676769] LNet: Removed LNI 192.168.206.119@tcp [12031.466417] Key type .llcrypt unregistered [12031.468880] Key type ._llcrypt unregistered