[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 572570632 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002339] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006769] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010114] LSM: Security Framework initializing [ 0.011042] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013056] *** VALIDATE selinux *** [ 0.020714] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025015] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027078] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030081] *** VALIDATE tmpfs *** [ 0.031484] *** VALIDATE proc *** [ 0.033039] *** VALIDATE cgroup *** [ 0.033954] *** VALIDATE cgroup2 *** [ 0.035196] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037016] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039025] Spectre V2 : User space: Vulnerable [ 0.040006] Speculative Store Bypass: Vulnerable [ 0.043874] debug: unmapping init [mem 0xffffffffbe259000-0xffffffffbe260fff] [ 0.045885] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046606] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047020] ... version: 2 [ 0.048008] ... bit width: 48 [ 0.049007] ... generic registers: 4 [ 0.049998] ... value mask: 0000ffffffffffff [ 0.051010] ... max period: 00007fffffffffff [ 0.052009] ... fixed-purpose events: 3 [ 0.053009] ... event mask: 000000070000000f [ 0.054251] rcu: Hierarchical SRCU implementation. [ 0.056317] smp: Bringing up secondary CPUs ... [ 0.057498] x86: Booting SMP configuration: [ 0.058018] .... node #0, CPUs: #1 #2 #3 [ 0.061522] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237576] node 0 deferred pages initialised in 169ms [ 0.240013] devtmpfs: initialized [ 0.242306] x86/mm: Memory block size: 128MB [ 0.246254] gcov: version magic: 0x41383552 [ 0.250379] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.251088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252439] pinctrl core: initialized pinctrl subsystem [ 0.255289] [ 0.255807] ************************************************************* [ 0.258016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261015] ** ** [ 0.264019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.267017] ** ** [ 0.269013] ** This means that this kernel is built to expose internal ** [ 0.272018] ** IOMMU data structures, which may compromise security on ** [ 0.274014] ** your system. ** [ 0.277017] ** ** [ 0.280017] ** If you see this message and you are not debugging the ** [ 0.282016] ** kernel, report this immediately to your vendor! ** [ 0.284023] ** ** [ 0.287016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.289015] ************************************************************* [ 0.291505] NET: Registered protocol family 16 [ 0.293384] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.296076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.299092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.302139] cpuidle: using governor menu [ 0.303687] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.307442] PCI: Using configuration type 1 for base access [ 0.310153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.320203] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.323043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.327065] cryptd: max_cpu_qlen set to 1000 [ 0.329263] ACPI: Added _OSI(Module Device) [ 0.331247] ACPI: Added _OSI(Processor Device) [ 0.333016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.335020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.340564] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.346415] ACPI: Interpreter enabled [ 0.348060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.349015] ACPI: Using IOAPIC for interrupt routing [ 0.351092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.355407] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.366360] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.368056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.371018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.375082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.380576] acpiphp: Slot [2] registered [ 0.382132] acpiphp: Slot [5] registered [ 0.384319] acpiphp: Slot [6] registered [ 0.386128] acpiphp: Slot [7] registered [ 0.387135] acpiphp: Slot [8] registered [ 0.388150] acpiphp: Slot [9] registered [ 0.390126] acpiphp: Slot [10] registered [ 0.392159] acpiphp: Slot [3] registered [ 0.394112] acpiphp: Slot [4] registered [ 0.396118] acpiphp: Slot [11] registered [ 0.397096] acpiphp: Slot [12] registered [ 0.398116] acpiphp: Slot [13] registered [ 0.400114] acpiphp: Slot [14] registered [ 0.401132] acpiphp: Slot [15] registered [ 0.403108] acpiphp: Slot [16] registered [ 0.405098] acpiphp: Slot [17] registered [ 0.406110] acpiphp: Slot [18] registered [ 0.408096] acpiphp: Slot [19] registered [ 0.409126] acpiphp: Slot [20] registered [ 0.411203] acpiphp: Slot [21] registered [ 0.413141] acpiphp: Slot [22] registered [ 0.414106] acpiphp: Slot [23] registered [ 0.415085] acpiphp: Slot [24] registered [ 0.417109] acpiphp: Slot [25] registered [ 0.419097] acpiphp: Slot [26] registered [ 0.420105] acpiphp: Slot [27] registered [ 0.421103] acpiphp: Slot [28] registered [ 0.422073] acpiphp: Slot [29] registered [ 0.424080] acpiphp: Slot [30] registered [ 0.425096] acpiphp: Slot [31] registered [ 0.426055] PCI host bridge to bus 0000:00 [ 0.427015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.429019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.431024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.434023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.436022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.439038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.441198] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.444491] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.447484] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.458925] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.463602] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.466017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.467014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.469021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.472585] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.476000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.477053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.480612] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.485017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.499022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.507989] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.514230] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.522019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.532022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.560020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.572674] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.584021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.598022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.615023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.631258] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.638018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.649021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.676022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.688866] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.696016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.703019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.719017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.728000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.737018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.748020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.769017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.783732] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.790015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.797016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.813018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.822614] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.826347] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.829327] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.832327] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.835241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.841164] iommu: Default domain type: Passthrough [ 0.842576] SCSI subsystem initialized [ 0.845159] ACPI: bus type USB registered [ 0.847243] usbcore: registered new interface driver usbfs [ 0.849093] usbcore: registered new interface driver hub [ 0.851076] usbcore: registered new device driver usb [ 0.852246] pps_core: LinuxPPS API ver. 1 registered [ 0.854011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.857113] PTP clock support registered [ 0.859171] EDAC MC: Ver: 3.0.0 [ 0.861121] PCI: Using ACPI for IRQ routing [ 0.862922] NetLabel: Initializing [ 0.864013] NetLabel: domain hash size = 128 [ 0.865008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.867079] NetLabel: unlabeled traffic allowed by default [ 0.869135] vgaarb: loaded [ 0.870334] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.872012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.879389] clocksource: Switched to clocksource kvm-clock [ 0.995525] VFS: Disk quotas dquot_6.6.0 [ 0.998081] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.002106] *** VALIDATE ramfs *** [ 1.003920] *** VALIDATE hugetlbfs *** [ 1.006126] pnp: PnP ACPI init [ 1.009378] pnp: PnP ACPI: found 6 devices [ 1.028824] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.033777] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.037010] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.039467] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.042380] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.045402] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.048655] NET: Registered protocol family 2 [ 1.051342] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.056382] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.060964] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.066592] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.071403] TCP: Hash tables configured (established 65536 bind 65536) [ 1.074764] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.078580] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.081744] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.085199] NET: Registered protocol family 1 [ 1.088241] RPC: Registered named UNIX socket transport module. [ 1.090576] RPC: Registered udp transport module. [ 1.092660] RPC: Registered tcp transport module. [ 1.094406] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.096914] NET: Registered protocol family 44 [ 1.098554] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.100767] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.102977] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.105189] PCI: CLS 0 bytes, default 64 [ 1.106649] Unpacking initramfs... [ 2.570806] debug: unmapping init [mem 0xffff8b2ffcc54000-0xffff8b2ffffbffff] [ 2.579240] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.581467] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.585902] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.106697] Initialise system trusted keyrings [ 3.108561] Key type blacklist registered [ 3.110804] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.120031] zbud: loaded [ 3.123283] *** VALIDATE nfs *** [ 3.124671] *** VALIDATE nfs4 *** [ 3.126511] pstore: using deflate compression [ 3.130605] Platform Keyring initialized [ 3.256031] NET: Registered protocol family 38 [ 3.257850] Key type asymmetric registered [ 3.259829] Asymmetric key parser 'x509' registered [ 3.261797] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.265157] io scheduler mq-deadline registered [ 3.267337] io scheduler kyber registered [ 3.269589] io scheduler bfq registered [ 3.272274] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.276911] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.281184] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.285379] ACPI: Power Button [PWRF] [ 3.291631] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.299453] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.319367] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.326793] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.345728] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.375322] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.407692] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.413963] Non-volatile memory driver v1.3 [ 3.415604] Linux agpgart interface v0.103 [ 3.451091] virtio_blk virtio1: [vda] 149968 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.453426] vda: detected capacity change from 0 to 76783616 [ 3.472111] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.477185] vdb: detected capacity change from 0 to 1073741824 [ 3.500513] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.503733] vdc: detected capacity change from 0 to 2621440000 [ 3.520432] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.524223] vdd: detected capacity change from 0 to 2621440000 [ 3.539120] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.542220] vde: detected capacity change from 0 to 4294967296 [ 3.558553] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.561454] vdf: detected capacity change from 0 to 4294967296 [ 3.569615] libphy: Fixed MDIO Bus: probed [ 3.575473] usbcore: registered new interface driver usbserial_generic [ 3.577645] usbserial: USB Serial support registered for generic [ 3.580104] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.584581] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.586319] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.588827] mousedev: PS/2 mouse device common for all mice [ 3.591599] rtc_cmos 00:05: RTC can wake from S4 [ 3.594468] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.594816] rtc_cmos 00:05: registered as rtc0 [ 3.600037] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.602438] intel_pstate: CPU model not supported [ 3.605311] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.606426] hid: raw HID events driver (C) Jiri Kosina [ 3.612827] usbcore: registered new interface driver usbhid [ 3.614958] usbhid: USB HID core driver [ 3.616926] drop_monitor: Initializing network drop monitor service [ 3.617169] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.619125] Initializing XFRM netlink socket [ 3.623813] NET: Registered protocol family 10 [ 3.626742] Segment Routing with IPv6 [ 3.628191] NET: Registered protocol family 17 [ 3.630258] mpls_gso: MPLS GSO support [ 3.635945] RAS: Correctable Errors collector initialized. [ 3.638140] AVX version of gcm_enc/dec engaged. [ 3.640332] AES CTR mode by8 optimization enabled [ 3.727049] sched_clock: Marking stable (3726994291, 0)->(4565942697, -838948406) [ 3.731252] registered taskstats version 1 [ 3.735758] Loading compiled-in X.509 certificates [ 3.738504] zswap: loaded using pool lzo/zbud [ 3.768047] Key type big_key registered [ 3.783468] Key type encrypted registered [ 3.785270] ima: No TPM chip found, activating TPM-bypass! [ 3.787361] ima: Allocated hash algorithm: sha1 [ 3.789045] ima: No architecture policies found [ 3.790281] evm: Initialising EVM extended attributes: [ 3.792561] evm: security.selinux [ 3.793918] evm: security.ima [ 3.794713] evm: security.capability [ 3.795786] evm: HMAC attrs: 0x1 [ 3.798063] rtc_cmos 00:05: setting system clock to 2026-09-10 18:34:20 UTC (1789065260) [ 3.803939] debug: unmapping init [mem 0xffffffffbf203000-0xffffffffbf3fffff] [ 3.806414] debug: unmapping init [mem 0xffffffffbdf82000-0xffffffffbe258fff] [ 3.814146] Write protecting the kernel read-only data: 28672k [ 3.817602] debug: unmapping init [mem 0xffffffffbc603000-0xffffffffbc7fffff] [ 3.820334] debug: unmapping init [mem 0xffffffffbcf14000-0xffffffffbcffffff] [ 3.862769] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.872407] systemd[1]: Detected virtualization kvm. [ 3.875277] systemd[1]: Detected architecture x86-64. [ 3.878815] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.904258] systemd[1]: No hostname configured. [ 3.905568] systemd[1]: Set hostname to . [ 3.907498] random: systemd: uninitialized urandom read (16 bytes read) [ 3.909637] systemd[1]: Initializing machine ID from random generator. [ 3.968398] random: ln: uninitialized urandom read (6 bytes read) [ 4.049604] random: systemd: uninitialized urandom read (16 bytes read) [ 4.051923] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.056891] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.063148] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.722559] device-mapper: uevent: version 1.0.3 [ 4.725597] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.608606] virtio_net virtio0 ens2: renamed from eth0 [ 5.671478] random: fast init done [ 5.727611] scsi host0: ata_piix [ 5.750277] scsi host1: ata_piix [ 5.785659] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.788731] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.505191] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.301340] random: crng init done [ 10.304732] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.879772] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.118116] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.387413] SELinux: Disabled at runtime. [ 12.451169] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.460606] systemd[1]: Detected virtualization kvm. [ 12.462718] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.028045] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.032413] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.038900] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.043710] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.047493] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.058276] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.063772] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ 13.133683] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.549736] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.990904] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.039601] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.214465] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.235204] EDAC sbridge: Ver: 1.1.2 [ 15.780234] Key type dns_resolver registered [ 16.129430] NFS: Registering the id_resolver key type [ 16.132463] Key type id_resolver registered [ 16.134937] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg627-server login: [ 29.746197] hrtimer: interrupt took 4070843 ns [ 70.995229] libcfs: loading out-of-tree module taints kernel. [ 71.032850] Key type ._llcrypt registered [ 71.034855] Key type .llcrypt registered [ 71.160519] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 89.051965] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 90.551710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.571666] alg: No test for adler32 (adler32-zlib) [ 91.829825] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 92.707480] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 94.543193] Key type lgssc registered [ 96.199241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 113.308729] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.806497] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 168.575802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.625872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.874609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.918436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.029497] Lustre: lustre-MDT0000: new disk, initializing [ 170.125059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.142210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.322988] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.996862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.054682] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.085554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 188.090289] Lustre: Skipped 1 previous similar message [ 188.186764] Lustre: lustre-MDT0001: new disk, initializing [ 188.256031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 188.280754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 188.297675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.748340] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.662558] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 203.148514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.329242] Lustre: lustre-OST0000: new disk, initializing [ 203.332491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.336955] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.375978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.239249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.247964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.340591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 207.859274] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.186068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.318894] Lustre: lustre-OST0001: new disk, initializing [ 219.322585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.328901] Lustre: 9532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.381748] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 224.340623] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.277892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 225.285167] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 225.318355] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.911449] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.468550] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 243.426322] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing check_logdir /tmp/testlogs/ [ 247.417919] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing yml_node [ 251.028437] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 252.810337] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 254.663727] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 255.749147] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 14:38:31 EDT 2026 [ 268.646127] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 269.678552] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 285.664180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 285.673180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.688912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.689450] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.690620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 286.702404] Lustre: Skipped 1 previous similar message [ 286.721894] Lustre: Skipped 3 previous similar messages [ 290.341277] Lustre: server umount lustre-MDT0000 complete [ 293.424875] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065550 with bad export cookie 13599880473708339950 [ 293.426352] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.431990] LustreError: 6514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 296.932343] LustreError: 6529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.935245] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.946891] LustreError: 6529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 296.975604] Lustre: Skipped 1 previous similar message [ 296.986046] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 296.994903] Lustre: Skipped 1 previous similar message [ 298.975761] LustreError: 6530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.004389] LustreError: 6530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 299.740466] Lustre: server umount lustre-MDT0001 complete [ 302.836313] Lustre: server umount lustre-OST0000 complete [ 305.738082] Lustre: server umount lustre-OST0001 complete [ 310.867487] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 319.060263] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 350.284332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.533399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 350.578801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 350.708493] Lustre: lustre-MDT0000: new disk, initializing [ 350.850935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 350.865507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 355.515244] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 365.999571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.183953] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 366.247244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 366.253631] Lustre: Skipped 1 previous similar message [ 366.392763] Lustre: lustre-MDT0001: new disk, initializing [ 366.544992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 366.578144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 366.593900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 371.289183] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 381.264488] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 386.289971] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 392.305849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 392.601627] Lustre: lustre-OST0000: new disk, initializing [ 392.605366] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 392.617612] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 392.697874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.251966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 394.266758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 394.435302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 398.380250] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 407.224391] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 411.175161] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 411.388401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 415.598574] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 415.796647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 419.809374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 419.816518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.830819] Lustre: Skipped 1 previous similar message [ 419.835278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 419.839724] Lustre: Skipped 2 previous similar messages [ 423.540090] Lustre: server umount lustre-OST0000 complete [ 428.006719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 428.018738] LustreError: Skipped 1 previous similar message [ 428.023964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.038425] Lustre: Skipped 1 previous similar message [ 428.048727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 428.059177] Lustre: Skipped 2 previous similar messages [ 432.949327] Lustre: server umount lustre-MDT0000 complete [ 435.681261] LustreError: 15093:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.700880] LustreError: 15093:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 436.063309] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065692 with bad export cookie 13599880473708341483 [ 436.066489] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 436.075822] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 436.375770] Lustre: server umount lustre-MDT0001 complete [ 440.975838] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 14:41:36 (1789065696) [ 447.431577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.126425] LustreError: 19312:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.201806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 453.266690] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 453.600845] LustreError: 19313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 461.525091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.531279] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 474.644385] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.382320] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 487.797741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 494.692788] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 502.660315] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 509.795515] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 516.281682] Lustre: server umount lustre-MDT0000 complete [ 517.089611] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.094525] LustreError: 21609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.104335] Lustre: Skipped 2 previous similar messages [ 517.130098] LustreError: 21609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 519.829954] LustreError: 21844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065776 with bad export cookie 13599880473708342365 [ 519.831283] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 519.855330] LustreError: 21844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 522.337905] Lustre: server umount lustre-MDT0001 complete [ 526.335750] Lustre: server umount lustre-OST0000 complete [ 547.419156] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 555.670082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.038692] LustreError: 23693:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.051076] LustreError: 23693:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 556.082351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.087142] Lustre: Skipped 2 previous similar messages [ 559.826122] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 567.573753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.057924] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 574.974328] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 581.723053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.648593] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 596.144737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 596.315354] Lustre: lustre-OST0001: new disk, initializing [ 596.318907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 596.324659] Lustre: 26008:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 598.339257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 598.358508] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 598.487703] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 602.090230] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 611.100163] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 614.592653] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 626.146180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.163220] Lustre: Skipped 3 previous similar messages [ 626.166717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 626.180061] Lustre: Skipped 4 previous similar messages [ 632.344290] Lustre: server umount lustre-MDT0000 complete [ 635.706584] LustreError: 23673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065892 with bad export cookie 13599880473708343212 [ 635.724443] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.195330] Lustre: server umount lustre-MDT0001 complete [ 649.598564] Lustre: server umount lustre-OST0000 complete [ 652.576105] Lustre: 3656:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789065893/real 1789065893] req@ffff8b2f45151880 x1875970987618432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789065909 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 653.004407] Lustre: server umount lustre-OST0001 complete [ 666.844475] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 676.515597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.942873] LustreError: 29030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.972165] LustreError: 29030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 680.919889] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 688.882394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.358494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 689.367651] Lustre: Skipped 4 previous similar messages [ 694.005887] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 696.577887] Lustre: 30179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 702.346472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 708.367845] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 717.418366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 724.829767] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 732.267384] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 735.652954] Lustre: 32060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 742.372375] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.381449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.389330] Lustre: Skipped 2 previous similar messages [ 742.406713] Lustre: Skipped 7 previous similar messages [ 748.262583] Lustre: server umount lustre-MDT0000 complete [ 751.911868] LustreError: 30561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066008 with bad export cookie 13599880473708344507 [ 751.933568] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 751.935647] LustreError: 30561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 752.499486] Lustre: server umount lustre-MDT0001 complete [ 766.835781] Lustre: server umount lustre-OST0000 complete [ 768.991786] Lustre: 3654:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789066009/real 1789066009] req@ffff8b2f450aad80 x1875970987666944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789066025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 770.639900] Lustre: server umount lustre-OST0001 complete [ 777.944434] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 14:47:13 (1789066033) [ 804.849878] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 814.338274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.709271] LustreError: 34860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 814.727252] LustreError: 34860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 818.765693] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 826.853757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 831.406230] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 833.962239] Lustre: 36007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 840.232499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 845.889902] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 854.057332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 861.737449] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 869.180887] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 873.122461] Lustre: 37889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 880.100980] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.103861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 880.125994] Lustre: Skipped 6 previous similar messages [ 880.142985] Lustre: Skipped 6 previous similar messages [ 885.312923] Lustre: server umount lustre-MDT0000 complete [ 888.932247] LustreError: 34841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066145 with bad export cookie 13599880473708345613 [ 888.941113] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 888.943884] LustreError: 34841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 889.257281] Lustre: server umount lustre-MDT0001 complete [ 903.257647] Lustre: server umount lustre-OST0000 complete [ 907.231313] Lustre: 3656:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789066147/real 1789066147] req@ffff8b2f4290d880 x1875970987715712/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789066163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 907.801584] Lustre: server umount lustre-OST0001 complete [ 916.436764] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 14:49:31 (1789066171) [ 932.292988] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 942.196659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.089761] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 947.682583] LustreError: 40031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 947.699715] LustreError: 40031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 955.270200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.594969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 955.603349] Lustre: Skipped 7 previous similar messages [ 959.433939] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 962.494573] Lustre: 41181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 968.913076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 976.445512] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 984.318952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.587109] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 998.974215] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1002.137763] Lustre: 43061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1002.979795] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1002.984468] Lustre: Skipped 1 previous similar message [ 1006.562946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1006.583633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.612720] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.633179] Lustre: Skipped 4 previous similar messages [ 1011.839982] Lustre: server umount lustre-MDT0000 complete [ 1015.050894] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066271 with bad export cookie 13599880473708346712 [ 1015.067268] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1015.076475] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1021.542128] Lustre: server umount lustre-MDT0001 complete [ 1029.753346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.319348] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1050.034648] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1053.833365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.622808] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1059.323868] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1059.330241] Lustre: Skipped 1 previous similar message [ 1065.775564] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1067.177900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1073.253979] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1075.284662] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1084.900906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1086.715477] Lustre: server umount lustre-MDT0000 complete [ 1090.159958] LustreError: 40012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066346 with bad export cookie 13599880473708347846 [ 1090.166237] LustreError: 40012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1090.172213] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1090.534936] Lustre: server umount lustre-MDT0001 complete [ 1103.532644] Lustre: server umount lustre-OST0000 complete [ 1116.644770] Lustre: server umount lustre-OST0001 complete [ 1125.122218] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 14:53:00 (1789066380) [ 1138.993425] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1147.930657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.582888] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1159.804401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.583404] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1167.410551] Lustre: 48815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1173.626755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1179.835748] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1187.409876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1193.708614] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1200.770461] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1204.719558] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1216.486105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1216.495775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1216.501141] Lustre: Skipped 14 previous similar messages [ 1218.018249] Lustre: server umount lustre-MDT0000 complete [ 1219.557223] LustreError: 48400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.582031] LustreError: 48400:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 30 previous similar messages [ 1221.438272] LustreError: 47645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066478 with bad export cookie 13599880473708348644 [ 1221.440880] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.454819] LustreError: 47645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1221.814814] Lustre: server umount lustre-MDT0001 complete [ 1236.026547] Lustre: server umount lustre-OST0000 complete [ 1249.570191] Lustre: server umount lustre-OST0001 complete [ 1258.193779] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 14:55:13 (1789066513) [ 1260.266200] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1261.635728] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 14:55:16 (1789066516) [ 1271.418673] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 1279.278244] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1296.853408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1297.072583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1297.171244] Lustre: lustre-MDT0000: new disk, initializing [ 1297.251215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1300.514096] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1308.478332] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1314.740353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1314.964632] Lustre: lustre-OST0000: new disk, initializing [ 1314.968959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1314.974715] Lustre: Skipped 1 previous similar message [ 1314.979291] Lustre: 54892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1316.926361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1316.931905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1316.956889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1320.393768] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1328.576775] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1330.375405] Lustre: DEBUG MARKER: create test files [ 1357.797548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1357.801281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.814537] Lustre: Skipped 10 previous similar messages [ 1361.164324] Lustre: server umount lustre-OST0000 complete [ 1364.600361] Lustre: server umount lustre-MDT0000 complete [ 1374.118631] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1376.791899] Key type lgssc unregistered [ 1377.079265] LNet: 56516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.088123] LNetError: 56516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.107891] LNet: Removed LNI 192.168.206.127@tcp [ 1378.042170] Key type .llcrypt unregistered [ 1378.045439] Key type ._llcrypt unregistered [ 1401.895583] Key type ._llcrypt registered [ 1401.898119] Key type .llcrypt registered [ 1401.982902] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1402.530677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1402.580166] alg: No test for adler32 (adler32-zlib) [ 1403.535941] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 1403.697515] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1405.335276] Key type lgssc registered [ 1406.278236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.682671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.708907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.362420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.238583] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1427.760431] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1434.440726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.693627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.877135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:98 to 0x240000400:129) [ 1441.000039] Lustre: 59226:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1441.017351] Lustre: 59226:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1441.024937] Lustre: 59226:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1441.036686] Lustre: 59226:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1441.051201] Lustre: 59226:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1441.068431] Lustre: 59226:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1441.343765] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1448.538816] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1454.482908] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1460.997432] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1470.317105] Lustre: server umount lustre-OST0000 complete [ 1483.945942] Lustre: server umount lustre-MDT0000 complete [ 1493.635480] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1496.131884] Key type lgssc unregistered [ 1496.390505] LNet: 60836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.398306] LNetError: 60836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.416138] LNet: Removed LNI 192.168.206.127@tcp [ 1497.131379] Key type .llcrypt unregistered [ 1497.135082] Key type ._llcrypt unregistered [ 1524.494842] Key type ._llcrypt registered [ 1524.499131] Key type .llcrypt registered [ 1524.596450] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1525.516486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1525.536286] alg: No test for adler32 (adler32-zlib) [ 1526.562773] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 1526.886047] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1528.544242] Key type lgssc registered [ 1529.518813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.895700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1536.909331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.217255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.474590] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1549.163328] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1555.270145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1555.557462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.336292] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1562.738688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1562.817382] Lustre: 63641:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1562.830103] Lustre: 63641:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1562.847236] Lustre: 63641:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1562.857971] Lustre: 63641:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1562.864641] Lustre: 63641:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1562.869263] Lustre: 63641:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1567.915477] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1572.859929] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1583.592148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1583.597969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.614116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.165864] Lustre: server umount lustre-OST0000 complete [ 1591.870765] Lustre: server umount lustre-MDT0000 complete [ 1605.299449] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1608.179442] Key type lgssc unregistered [ 1608.490658] LNet: 65210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.497923] LNetError: 65210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.513881] LNet: Removed LNI 192.168.206.127@tcp [ 1610.514163] Key type .llcrypt unregistered [ 1610.518707] Key type ._llcrypt unregistered [ 1627.315408] Key type ._llcrypt registered [ 1627.319932] Key type .llcrypt registered [ 1627.433413] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 1643.566588] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1645.046049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1645.092207] alg: No test for adler32 (adler32-zlib) [ 1646.399721] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 1646.811158] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1648.575185] Key type lgssc registered [ 1650.035856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1685.913731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1685.957866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.126970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1687.146396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1687.287893] Lustre: lustre-MDT0000: new disk, initializing [ 1687.404133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.421765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1692.499122] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1703.813406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1703.965395] Lustre: 68366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1704.044454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1704.049428] Lustre: Skipped 1 previous similar message [ 1704.142610] Lustre: lustre-MDT0001: new disk, initializing [ 1704.286676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1704.331532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1704.355358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1709.088031] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1717.393849] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1722.439663] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1730.080655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1730.481068] Lustre: lustre-OST0000: new disk, initializing [ 1730.488702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1730.500925] Lustre: 70332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1730.606711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.096747] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1739.329671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1739.341592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1739.456136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1746.235968] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1750.613659] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1750.787300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1755.578273] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1755.737628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1760.226845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.238408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1760.271466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.286549] Lustre: Skipped 1 previous similar message [ 1763.642363] Lustre: server umount lustre-OST0000 complete [ 1770.465548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.466181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.486856] Lustre: Skipped 2 previous similar messages [ 1770.506274] Lustre: Skipped 1 previous similar message [ 1775.584693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1775.602273] Lustre: Skipped 1 previous similar message [ 1780.709539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1780.714800] Lustre: Skipped 1 previous similar message [ 1781.215121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1781.348936] Lustre: server umount lustre-MDT0000 complete [ 1784.899873] LustreError: 68357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067041 with bad export cookie 4359720165060611068 [ 1784.902983] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1784.910380] LustreError: 68357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1785.385344] Lustre: server umount lustre-MDT0001 complete [ 1793.498457] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 15:04:08 (1789067048) [ 1803.942871] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1842.359663] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 1851.006716] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1886.933651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.166251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1887.186419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1887.242696] Lustre: lustre-MDT0000: new disk, initializing [ 1887.318797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.338182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1891.805500] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1901.681031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.861038] Lustre: 76353:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1901.936618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1901.946940] Lustre: Skipped 1 previous similar message [ 1902.029982] Lustre: lustre-MDT0001: new disk, initializing [ 1902.140673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1902.182995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1902.193024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1906.556407] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1915.120574] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1920.208375] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1926.870691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.189796] Lustre: lustre-OST0000: new disk, initializing [ 1927.193712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1927.198591] Lustre: 78323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1927.256226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1934.926748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1934.940615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1935.113129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1935.380867] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1948.109756] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1952.705651] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1952.994567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1956.753948] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1956.963817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1960.935025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1960.939429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1960.946595] LustreError: Skipped 1 previous similar message [ 1960.947523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1960.956934] Lustre: Skipped 1 previous similar message [ 1960.969241] Lustre: Skipped 1 previous similar message [ 1971.169746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1971.186435] Lustre: Skipped 3 previous similar messages [ 1973.215838] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1973.362438] Lustre: server umount lustre-OST0000 complete [ 1978.851621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1978.866823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.700949] Lustre: server umount lustre-MDT0000 complete [ 1986.559553] LustreError: 78322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067243 with bad export cookie 4359720165060611950 [ 1986.572253] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1986.577189] LustreError: 78322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1986.885863] Lustre: server umount lustre-MDT0001 complete [ 1995.257640] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 15:07:30 (1789067250) [ 2001.393596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.841269] LustreError: 80738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.877937] LustreError: 80738:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2001.957937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2006.188812] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2007.012265] LustreError: 80739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.132947] LustreError: 80738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.956866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.369763] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2025.177084] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2031.469613] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2038.775358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2039.104796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.118700] Lustre: Skipped 1 previous similar message [ 2045.746841] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2053.601362] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2057.400739] Lustre: 83408:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting 4d3b81de-3102-4a20-83ed-37a90648f3ad at adminstrative request [ 2061.798981] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.834159] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2072.045385] Lustre: MGS: Client 4d3b81de-3102-4a20-83ed-37a90648f3ad (at 0@lo) reconnecting [ 2072.055254] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x3c80d6642e8b92e0 to 0x3c80d6642e8b9580 [ 2072.090531] Lustre: MGC192.168.206.127@tcp: Connection restored to 0@lo (at 0@lo) [ 2093.817917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.988340] Lustre: lustre-OST0001: new disk, initializing [ 2093.992367] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2094.001213] Lustre: 83802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2099.353565] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2101.357189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2101.366979] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2101.479875] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2107.216295] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2111.972755] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2111.981699] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.991654] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2111.991663] Lustre: Skipped 3 previous similar messages [ 2112.002653] Lustre: Skipped 2 previous similar messages [ 2115.870515] Lustre: server umount lustre-OST0001 complete [ 2117.099180] LustreError: 82559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.135357] LustreError: 82559:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2122.208505] LustreError: 82558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.209620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.249192] LustreError: 82558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2122.273829] Lustre: Skipped 1 previous similar message [ 2122.289659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2122.302210] LustreError: Skipped 1 previous similar message [ 2126.076884] Lustre: server umount lustre-OST0000 complete [ 2131.937096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2136.470523] Lustre: server umount lustre-MDT0000 complete [ 2137.573098] LustreError: 80739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.596683] LustreError: 80739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2140.220360] LustreError: 83801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067396 with bad export cookie 4359720165060613504 [ 2140.229363] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2140.240325] LustreError: 83801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2140.684530] Lustre: server umount lustre-MDT0001 complete [ 2148.518899] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 15:10:03 (1789067403) [ 2150.099622] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2151.845772] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 15:10:06 (1789067406) [ 2153.197503] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2154.992586] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 15:10:09 (1789067409) [ 2156.606527] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2158.410796] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 15:10:13 (1789067413) [ 2159.736734] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2161.558635] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 15:10:16 (1789067416) [ 2163.067782] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2168.362330] loop: module loaded [ 2168.380557] loop0: detected capacity change from 0 to 4294967296 [ 2168.406417] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2170.196665] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2173.118968] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2176.912324] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2202.400317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2202.451658] LustreError: 87921:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2202.457637] LustreError: 87921:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2202.462747] LustreError: 87921:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2202.469209] LustreError: 87921:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2219.544372] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 2228.356846] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2259.318602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.571661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2259.597920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2259.671530] Lustre: lustre-MDT0000: new disk, initializing [ 2259.745584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2259.760360] Lustre: Skipped 1 previous similar message [ 2259.793682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2263.396684] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2272.697712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.933138] Lustre: 90789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2273.311724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2277.793294] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2288.430961] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2294.028572] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2301.236915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2301.618838] Lustre: 92759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2307.809515] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2308.207476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2308.359394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2317.346894] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2321.994449] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2322.198369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2326.408940] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2326.627884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2329.069047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2329.074537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.083458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2329.100066] Lustre: Skipped 3 previous similar messages [ 2329.115603] Lustre: Skipped 7 previous similar messages [ 2334.600518] Lustre: server umount lustre-OST0000 complete [ 2352.098530] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2352.256465] Lustre: server umount lustre-MDT0000 complete [ 2354.658437] LustreError: 90799:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.674727] LustreError: 90799:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2355.431316] LustreError: 90781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067612 with bad export cookie 4359720165060614113 [ 2355.431933] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2355.466366] LustreError: 90781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2355.769745] Lustre: server umount lustre-MDT0001 complete [ 2363.216189] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 15:13:38 (1789067618) [ 2369.665601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.936587] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2383.033949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.839549] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2394.489644] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2399.580275] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2407.378823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2407.852084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.861171] Lustre: Skipped 4 previous similar messages [ 2414.921782] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2422.910808] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2427.390873] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2427.649163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2433.456840] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2433.682720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2440.572959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2440.749849] Lustre: lustre-OST0001: new disk, initializing [ 2440.753281] Lustre: Skipped 2 previous similar messages [ 2440.755951] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2440.759714] Lustre: Skipped 3 previous similar messages [ 2440.762781] Lustre: 98539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2447.375757] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2450.021360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2450.034172] Lustre: Skipped 2 previous similar messages [ 2450.046934] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2450.157375] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2455.906924] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2459.869791] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2460.087666] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2464.065973] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2464.257709] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2483.160416] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2504.967939] Lustre: Failing over lustre-MDT0000 [ 2505.415443] Lustre: server umount lustre-MDT0000 complete [ 2506.209941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2506.225688] LustreError: Skipped 1 previous similar message [ 2506.235694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.256049] Lustre: Skipped 2 previous similar messages [ 2506.259830] LustreError: 95897:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.277836] LustreError: 95897:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2526.175323] Lustre: 66588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789067766/real 1789067766] req@00000000655b369b x1875972617152512/t0(0) o400->MGC192.168.206.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789067782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2526.202279] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2528.723991] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2528.727855] LDISKFS-fs (dm-0): recovery complete [ 2528.744295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2535.711455] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2539.325388] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2539.332496] Lustre: 96178:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2539.573721] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2541.037185] Lustre: 95170:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2541.051178] Lustre: 95170:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 2541.068345] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2541.135992] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2541.455127] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2541.778579] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2541.788847] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2542.111128] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2542.115024] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2542.123129] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2542.135918] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 7 previous similar messages [ 2543.095551] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2543.108606] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 3 previous similar messages [ 2543.431146] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2543.443072] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 3 previous similar messages [ 2544.411674] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2544.422527] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 6 previous similar messages [ 2545.107406] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2545.120177] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 5 previous similar messages [ 2545.455458] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2545.466750] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 5 previous similar messages [ 2548.633399] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2548.662629] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 2549.313097] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2549.324840] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2549.639116] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2549.643710] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2556.673383] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2556.686342] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 2557.332608] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2557.354564] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2557.679156] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2557.689816] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 2572.929420] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2572.939943] Lustre: 101382:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 50 previous similar messages [ 2573.588537] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2573.605827] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 2573.927407] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2573.935212] LustreError: 101382:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 2599.295201] Lustre: 101382:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2599.307251] Lustre: 101382:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2599.324696] Lustre: 101382:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d5a674fc-ff6e-477b-b4d6-9650dbe9d4d0@192.168.206.27@tcp [ 2599.340835] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2599.346916] Lustre: 101382:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@000000005b74d598 x1875972599125504/t0(8589934779) o101->d5a674fc-ff6e-477b-b4d6-9650dbe9d4d0@192.168.206.27@tcp:236/0 lens 592/0 e 0 to 0 dl 1789067866 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2599.375739] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2599.396986] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2599.404282] Lustre: lustre-MDT0000: Denying connection for new client d5a674fc-ff6e-477b-b4d6-9650dbe9d4d0 (at 192.168.206.27@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2599.446108] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2599.450324] Lustre: Skipped 2 previous similar messages [ 2599.459401] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2599.501544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2599.503851] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2615.787101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.792134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.805617] Lustre: Skipped 3 previous similar messages [ 2615.815147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.822548] Lustre: Skipped 8 previous similar messages [ 2628.064575] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2628.207685] Lustre: server umount lustre-OST0000 complete [ 2633.190899] LustreError: 96991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.231223] LustreError: 96991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 2637.676957] Lustre: server umount lustre-OST0001 complete [ 2655.199248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2655.408989] Lustre: server umount lustre-MDT0000 complete [ 2659.076130] LustreError: 95149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067915 with bad export cookie 4359720165060665332 [ 2659.083194] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2659.086571] LustreError: 95149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2659.456196] Lustre: server umount lustre-MDT0001 complete [ 2666.462839] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 15:18:41 (1789067921) [ 2672.678124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2672.685297] Lustre: *** cfs_fail_loc=197, val=0*** [ 2688.287571] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2693.407531] LustreError: 103362:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.127@tcp: failed processing log, type 4: rc = -110 [ 2698.628368] Lustre: *** cfs_fail_loc=197, val=0*** [ 2698.635248] Lustre: Skipped 65 previous similar messages [ 2719.200609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.203818] Lustre: Skipped 2 previous similar messages [ 2725.450565] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2731.866570] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2733.870777] Lustre: server umount lustre-OST0000 complete [ 2741.600729] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 15:19:56 (1789067996) [ 2760.784956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2764.772924] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2766.312213] LustreError: 104983:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.339110] LustreError: 104983:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2774.021951] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2777.998660] LustreError: 105013:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2778.008736] LustreError: 105013:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2778.019675] LustreError: 105013:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2778.499452] Lustre: server umount lustre-MDT0000 complete [ 2810.704310] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 2819.712783] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2851.721685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.974678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2851.997239] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2852.079424] Lustre: lustre-MDT0000: new disk, initializing [ 2852.161407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2855.979299] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2865.404572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.515164] Lustre: 109184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2865.711570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2869.819696] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2878.577955] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2883.231124] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2889.038365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.243825] Lustre: 111152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2894.960445] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2895.015728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2903.486879] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2907.946271] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2908.179475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2912.174203] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2912.336351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2915.296459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2915.315682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2915.337835] Lustre: Skipped 5 previous similar messages [ 2915.344412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2915.351581] Lustre: Skipped 14 previous similar messages [ 2919.950937] Lustre: server umount lustre-OST0000 complete [ 2929.909848] Lustre: server umount lustre-MDT0000 complete [ 2933.224022] LustreError: 109177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068189 with bad export cookie 4359720165060676007 [ 2933.232712] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2933.244346] LustreError: 109177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2933.710479] Lustre: server umount lustre-MDT0001 complete [ 2941.284241] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 15:23:16 (1789068196) [ 2949.585910] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 2952.857860] Key type lgssc unregistered [ 2953.257166] LNet: 113640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2953.262483] LNetError: 113640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2953.276939] LNet: Removed LNI 192.168.206.127@tcp [ 2954.099632] Key type .llcrypt unregistered [ 2954.102108] Key type ._llcrypt unregistered [ 2971.522653] Key type ._llcrypt registered [ 2971.525272] Key type .llcrypt registered [ 2971.631260] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 2983.726197] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2984.366468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2984.455041] alg: No test for adler32 (adler32-zlib) [ 2985.525072] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 2985.752112] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 2987.439268] Key type lgssc registered [ 2988.554830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3025.356731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3025.390635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.596839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3026.630321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3026.704715] Lustre: lustre-MDT0000: new disk, initializing [ 3026.774357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.789606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3030.266808] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3039.633551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.923837] Lustre: lustre-OST0000: new disk, initializing [ 3039.926491] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3039.929927] Lustre: Skipped 1 previous similar message [ 3039.937511] Lustre: 117788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3040.004408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3045.261490] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3048.026705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3048.033185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3048.069346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3055.298334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3055.509067] Lustre: lustre-OST0001: new disk, initializing [ 3055.521193] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3055.534977] Lustre: 118650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3055.651792] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3057.023482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3057.028715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3057.063040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3061.766285] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3072.483678] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3072.485337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3072.497367] Lustre: Skipped 1 previous similar message [ 3072.518974] Lustre: Skipped 1 previous similar message [ 3075.675689] Lustre: server umount lustre-MDT0000 complete [ 3081.402424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3086.911130] LustreError: 116976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068343 with bad export cookie 8212100476411302809 [ 3086.921063] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3086.928567] LustreError: 116976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3086.995544] Lustre: server umount lustre-OST0000 complete [ 3091.694501] Lustre: server umount lustre-OST0001 complete [ 3102.046495] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 15:25:56 (1789068356) [ 3122.157495] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 15:26:17 (1789068377) [ 3130.935288] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3151.022987] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 15:26:45 (1789068405) [ 3184.561904] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 3192.944644] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3224.631556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.875135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3224.908427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3225.032456] Lustre: lustre-MDT0000: new disk, initializing [ 3225.117314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.134267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3229.353167] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3238.993191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.160960] Lustre: 126401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3239.217597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3239.222176] Lustre: Skipped 1 previous similar message [ 3239.284503] Lustre: lustre-MDT0001: new disk, initializing [ 3239.364614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3239.406410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3239.423812] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3243.001244] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3251.592128] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3256.614197] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3263.872563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.127785] Lustre: lustre-OST0000: new disk, initializing [ 3264.132436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3264.140432] Lustre: 128369:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3264.219329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.632627] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3270.682897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3270.691335] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3270.781233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3280.141231] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3284.370598] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3284.529827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3289.117389] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3289.294126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3291.107837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3291.110173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3291.136400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3291.146257] Lustre: Skipped 1 previous similar message [ 3296.225264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3296.240343] Lustre: Skipped 1 previous similar message [ 3297.205679] Lustre: server umount lustre-OST0000 complete [ 3301.345263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3301.358672] Lustre: Skipped 1 previous similar message [ 3301.361352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3306.465033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3306.471022] Lustre: Skipped 2 previous similar messages [ 3307.280908] Lustre: server umount lustre-MDT0000 complete [ 3310.817504] LustreError: 126392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068567 with bad export cookie 8212100476411304034 [ 3310.819278] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3310.828439] LustreError: 126392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3311.217528] Lustre: server umount lustre-MDT0001 complete [ 3319.877450] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 15:29:34 (1789068574) [ 3326.866864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.499447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3332.316716] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3332.580633] LustreError: 130784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.602087] LustreError: 130784:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3337.697840] LustreError: 130783:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.725395] LustreError: 130784:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.903953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.860282] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3351.322759] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3356.308560] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3363.001175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3363.264291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.272305] Lustre: Skipped 1 previous similar message [ 3369.445266] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3377.253741] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3381.332913] Lustre: *** cfs_fail_loc=159, val=0*** [ 3381.343774] Lustre: Skipped 3 previous similar messages [ 3396.990834] Lustre: lustre-MDT0000: Client 20e1026a-184e-4b04-b81d-e288cbb85eb0 (at 192.168.206.27@tcp) reconnecting [ 3398.492988] Lustre: *** cfs_fail_loc=159, val=0*** [ 3398.504710] Lustre: Skipped 2 previous similar messages [ 3413.880604] Lustre: lustre-MDT0000: Client 20e1026a-184e-4b04-b81d-e288cbb85eb0 (at 192.168.206.27@tcp) reconnecting [ 3419.621758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.624319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3419.639245] Lustre: Skipped 1 previous similar message [ 3419.640627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3420.564834] Lustre: server umount lustre-OST0000 complete [ 3424.738108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.759302] Lustre: Skipped 1 previous similar message [ 3430.450790] Lustre: server umount lustre-MDT0000 complete [ 3433.614117] LustreError: 131517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068690 with bad export cookie 8212100476411304916 [ 3433.632350] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3433.904261] Lustre: server umount lustre-MDT0001 complete [ 3443.652580] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3446.116778] Key type lgssc unregistered [ 3446.388839] LNet: 134552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3446.392964] LNetError: 134552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3446.408861] LNet: Removed LNI 192.168.206.127@tcp [ 3447.144881] Key type .llcrypt unregistered [ 3447.149549] Key type ._llcrypt unregistered [ 3469.469482] Key type ._llcrypt registered [ 3469.471394] Key type .llcrypt registered [ 3469.570624] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3470.599336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3470.653642] alg: No test for adler32 (adler32-zlib) [ 3471.716648] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 3471.926756] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3473.591820] Key type lgssc registered [ 3474.867340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3482.051356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3482.088893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.509632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.308981] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3493.865017] LustreError: 135965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3493.883659] LustreError: 135965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3495.679344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.107590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3500.423450] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3507.280216] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3512.930764] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3519.340665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3519.815883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.656432] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3529.093171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3534.876041] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3544.595034] Lustre: *** cfs_fail_loc=159, val=0*** [ 3559.815750] Lustre: lustre-MDT0000: Client 86e5435a-68c1-49d4-93a4-c531e4814e66 (at 192.168.206.27@tcp) reconnecting [ 3562.512718] Lustre: *** cfs_fail_loc=159, val=0*** [ 3577.732648] Lustre: lustre-MDT0001: Client 86e5435a-68c1-49d4-93a4-c531e4814e66 (at 192.168.206.27@tcp) reconnecting [ 3579.213321] Lustre: *** cfs_fail_loc=159, val=0*** [ 3579.220653] Lustre: Skipped 3 previous similar messages [ 3595.637399] Lustre: lustre-MDT0001: Client 86e5435a-68c1-49d4-93a4-c531e4814e66 (at 192.168.206.27@tcp) reconnecting [ 3600.347837] Lustre: 139012:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3602.829945] Lustre: *** cfs_fail_loc=159, val=0*** [ 3602.832763] Lustre: Skipped 4 previous similar messages [ 3619.217195] Lustre: lustre-MDT0000: Client bd75f759-757e-4ccb-97f8-0d29911997cb (at 192.168.206.27@tcp) reconnecting [ 3621.427610] Lustre: *** cfs_fail_loc=159, val=0*** [ 3621.431529] Lustre: Skipped 14 previous similar messages [ 3637.126330] Lustre: lustre-MDT0000: Client bd75f759-757e-4ccb-97f8-0d29911997cb (at 192.168.206.27@tcp) reconnecting [ 3638.662194] Lustre: 139304:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3659.233543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3659.243358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.262471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3662.318239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3662.319137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.350559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3663.855531] Lustre: server umount lustre-OST0000 complete [ 3670.508154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3670.534133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3670.546571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3670.549347] Lustre: Skipped 1 previous similar message [ 3672.546456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.551061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3672.575045] Lustre: Skipped 1 previous similar message [ 3673.795710] Lustre: server umount lustre-MDT0000 complete [ 3677.273397] LustreError: 135946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789068933 with bad export cookie 6430406752931242628 [ 3677.290574] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3677.296864] LustreError: 135946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3677.722857] Lustre: server umount lustre-MDT0001 complete [ 3687.715373] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3690.442454] Key type lgssc unregistered [ 3690.707377] LNet: 140382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3690.716534] LNetError: 140382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3691.758432] LNet: Removed LNI 192.168.206.127@tcp [ 3692.581055] Key type .llcrypt unregistered [ 3692.585078] Key type ._llcrypt unregistered [ 3716.449117] Key type ._llcrypt registered [ 3716.453506] Key type .llcrypt registered [ 3716.561306] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3717.886394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3717.907545] alg: No test for adler32 (adler32-zlib) [ 3719.005937] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 3719.242867] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3720.895264] Key type lgssc registered [ 3722.051553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3728.953481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3728.987095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.435523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3734.015300] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3740.643888] LustreError: 141794:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3740.654851] LustreError: 141794:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3742.201748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.504135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3746.613748] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3753.939990] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3758.927067] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3765.197043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.611323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3767.667995] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3771.315068] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3775.856983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3778.585345] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3784.385521] Lustre: 144373:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3788.274619] Lustre: 144421:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3789.940509] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3791.121677] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3794.850197] Lustre: 144568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3796.052250] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3797.047828] Lustre: 144667:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3801.573117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.578707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3801.590253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3801.597612] Lustre: Skipped 1 previous similar message [ 3801.627256] Lustre: Skipped 1 previous similar message [ 3806.695039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3806.699646] Lustre: Skipped 1 previous similar message [ 3806.972414] Lustre: server umount lustre-OST0000 complete [ 3811.808094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.815537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3811.822923] Lustre: Skipped 1 previous similar message [ 3816.506708] Lustre: server umount lustre-MDT0000 complete [ 3816.931922] LustreError: 141794:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.950650] LustreError: 141794:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3820.981970] LustreError: 141776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069077 with bad export cookie 9783018905198419471 [ 3820.989647] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3820.994923] LustreError: 141776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3821.424950] Lustre: server umount lustre-MDT0001 complete [ 3831.406624] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3834.171618] Key type lgssc unregistered [ 3834.469213] LNet: 145728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3834.476678] LNetError: 145728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3834.499273] LNet: Removed LNI 192.168.206.127@tcp [ 3835.361749] Key type .llcrypt unregistered [ 3835.364802] Key type ._llcrypt unregistered [ 3859.463878] Key type ._llcrypt registered [ 3859.466442] Key type .llcrypt registered [ 3859.612714] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3860.729266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3860.783644] alg: No test for adler32 (adler32-zlib) [ 3861.861171] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 3862.154364] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3863.855491] Key type lgssc registered [ 3865.098525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3873.391258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3873.429104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.875568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3880.117562] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3885.035111] LustreError: 147138:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3885.075058] LustreError: 147138:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3890.160918] LustreError: 147137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.469882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.571111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3897.158683] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3907.365289] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3914.345268] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3923.435688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.225248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3931.948123] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3935.471493] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 3940.565464] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3944.688310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 3950.298047] Lustre: *** cfs_fail_loc=159, val=0*** [ 3965.822357] Lustre: lustre-MDT0000: Client 07b0484a-ab0c-4b7f-9d08-5f374714a7da (at 192.168.206.27@tcp) reconnecting [ 3971.552548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3971.567143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.578510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3972.078678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3972.089414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.090045] LustreError: Skipped 1 previous similar message [ 3972.149239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3973.471360] Lustre: server umount lustre-OST0000 complete [ 3978.216682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3978.229275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.248824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3980.771566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3980.774964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.867172] Lustre: server umount lustre-MDT0000 complete [ 3985.890719] LustreError: 147869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.933790] LustreError: 147869:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3987.729978] LustreError: 147117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069244 with bad export cookie 5510255675073091076 [ 3987.746345] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.033782] Lustre: server umount lustre-MDT0001 complete [ 3999.181970] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 4002.064718] Key type lgssc unregistered [ 4002.377825] LNet: 150800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4002.385652] LNetError: 150800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4002.405753] LNet: Removed LNI 192.168.206.127@tcp [ 4003.296805] Key type .llcrypt unregistered [ 4003.300045] Key type ._llcrypt unregistered [ 4026.760987] Key type ._llcrypt registered [ 4026.763826] Key type .llcrypt registered [ 4026.844663] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4028.050156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4028.097431] alg: No test for adler32 (adler32-zlib) [ 4029.201311] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 4029.466516] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 4031.135187] Key type lgssc registered [ 4032.349479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4039.759276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4039.781327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.382461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4045.811231] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4051.940664] LustreError: 152212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.960372] LustreError: 152212:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4054.860529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4055.599726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4060.535393] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4068.271480] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4073.922775] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4081.195781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4081.863273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.049219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4089.510366] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4099.766097] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4100.345058] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4121.710094] Lustre: lustre-MDT0001: evicting 50bf6e06-3bbf-440d-ac31-61a9bbbcd62c (at 192.168.206.27@tcp) by administrative request [ 4121.717383] Lustre: Skipped 1 previous similar message [ 4151.778465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4151.780608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.785460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4151.805965] Lustre: Skipped 1 previous similar message [ 4156.066417] Lustre: server umount lustre-OST0000 complete [ 4162.019408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.026486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4162.052088] Lustre: Skipped 1 previous similar message [ 4162.092988] Lustre: Skipped 2 previous similar messages [ 4167.151212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4167.162673] Lustre: Skipped 1 previous similar message [ 4172.267578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.280625] Lustre: Skipped 1 previous similar message [ 4174.303481] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4174.878778] Lustre: server umount lustre-MDT0000 complete [ 4177.380481] LustreError: 152213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.687343] LustreError: 152194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069438 with bad export cookie 14678461932059363043 [ 4181.694120] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4181.704229] LustreError: 152194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4182.358585] Lustre: server umount lustre-MDT0001 complete [ 4194.738619] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 4198.278835] Key type lgssc unregistered [ 4198.631713] LNet: 156288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4198.651281] LNetError: 156288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4198.670615] LNet: Removed LNI 192.168.206.127@tcp [ 4199.967399] Key type .llcrypt unregistered [ 4199.972951] Key type ._llcrypt unregistered [ 4282.168469] Key type ._llcrypt registered [ 4282.171510] Key type .llcrypt registered [ 4282.301855] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4283.447411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4283.552635] alg: No test for adler32 (adler32-zlib) [ 4284.737898] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 4285.107124] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 4286.815170] Key type lgssc registered [ 4288.560823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4297.166382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4297.225667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4298.798643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.071109] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4308.961889] LustreError: 158774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.988801] LustreError: 158774:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4311.900302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.290568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4316.784267] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4324.585178] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4329.880896] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4337.203380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4337.733296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4343.860408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4346.389221] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4356.020439] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4357.185298] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4367.328996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4367.338050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.344913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4367.364670] Lustre: Skipped 1 previous similar message [ 4371.715555] Lustre: server umount lustre-OST0000 complete [ 4377.568934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.581774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4377.589552] Lustre: Skipped 1 previous similar message [ 4377.599568] Lustre: Skipped 2 previous similar messages [ 4381.788549] Lustre: server umount lustre-MDT0000 complete [ 4382.692194] LustreError: 158774:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.723366] LustreError: 158774:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4385.588296] LustreError: 158755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069642 with bad export cookie 9065850265690843870 [ 4385.592563] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4385.601659] LustreError: 158755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4386.081640] Lustre: server umount lustre-MDT0001 complete [ 4398.293648] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 4401.398369] Key type lgssc unregistered [ 4401.760363] LNet: 162314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4401.771458] LNetError: 162314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4401.811244] LNet: Removed LNI 192.168.206.127@tcp [ 4402.655838] Key type .llcrypt unregistered [ 4402.660414] Key type ._llcrypt unregistered [ 4425.451467] Key type ._llcrypt registered [ 4425.455469] Key type .llcrypt registered [ 4425.549709] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4426.234732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4426.297326] alg: No test for adler32 (adler32-zlib) [ 4427.672758] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 4427.863710] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 4429.537882] Key type lgssc registered [ 4431.041721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4446.581265] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 15:48:21 (1789069701) [ 4463.836164] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 4474.392356] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4511.134686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4511.149816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.577398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4512.653573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4512.793512] Lustre: lustre-MDT0000: new disk, initializing [ 4512.905823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.930055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4518.635693] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4531.536452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.673158] Lustre: 166592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4531.735978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4531.740275] Lustre: Skipped 1 previous similar message [ 4531.839386] Lustre: lustre-MDT0001: new disk, initializing [ 4531.954725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4532.006466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4532.017556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4536.339293] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4547.212951] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4553.576949] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4560.500354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4560.855711] Lustre: lustre-OST0000: new disk, initializing [ 4560.861332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4560.871430] Lustre: 168559:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4560.980383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.101596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4567.116596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4567.228267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4567.458840] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4576.856811] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4588.007897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.045197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4588.048659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4588.065243] Lustre: Skipped 1 previous similar message [ 4592.787450] Lustre: server umount lustre-OST0000 complete [ 4598.241638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4598.253201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4598.257434] Lustre: Skipped 2 previous similar messages [ 4602.548581] Lustre: server umount lustre-MDT0000 complete [ 4606.074148] LustreError: 168569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789069862 with bad export cookie 1521880510461254980 [ 4606.076527] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4606.085361] LustreError: 168569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4606.570947] Lustre: server umount lustre-MDT0001 complete [ 4618.080401] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 4621.140935] Key type lgssc unregistered [ 4621.502665] LNet: 170464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4621.510745] LNetError: 170464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4621.536720] LNet: Removed LNI 192.168.206.127@tcp [ 4622.494583] Key type .llcrypt unregistered [ 4622.497867] Key type ._llcrypt unregistered [ 4647.265275] Key type ._llcrypt registered [ 4647.267946] Key type .llcrypt registered [ 4647.399083] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4648.860920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4648.927119] alg: No test for adler32 (adler32-zlib) [ 4650.163464] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 4650.399449] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 4652.098471] Key type lgssc registered [ 4653.072633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4661.013312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4661.027051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.503810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.816367] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4672.992767] LustreError: 171874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4673.014080] LustreError: 171874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4675.087637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.555145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4679.436279] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4686.515958] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4692.047468] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4698.697165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4699.245237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.318174] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4707.505861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4715.435511] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4722.203660] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4722.238193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4722.271680] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4722.276875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4722.286631] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4723.243142] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4723.273341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.290750] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4723.301256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4723.321062] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4724.176549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4725.109687] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4725.113562] Lustre: Skipped 1 previous similar message [ 4725.143715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.154717] Lustre: Skipped 1 previous similar message [ 4725.159770] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4725.169118] Lustre: Skipped 1 previous similar message [ 4725.196802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4725.212674] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4725.227097] Lustre: Skipped 1 previous similar message [ 4726.343273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4727.137625] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4727.142066] Lustre: Skipped 1 previous similar message [ 4727.160644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4727.169324] Lustre: Skipped 1 previous similar message [ 4727.174514] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4727.179280] Lustre: Skipped 1 previous similar message [ 4727.184089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.231631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.248762] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4728.252073] Lustre: Skipped 2 previous similar messages [ 4729.120823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4729.938499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4730.824643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4731.930465] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4731.938009] Lustre: Skipped 4 previous similar messages [ 4731.949952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.956592] Lustre: Skipped 4 previous similar messages [ 4731.959909] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4731.966518] Lustre: Skipped 4 previous similar messages [ 4731.969831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4733.174333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4733.196472] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4733.207409] Lustre: Skipped 4 previous similar messages [ 4734.144404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4735.007730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4735.774468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4736.793493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4737.639879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4738.443693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4739.328084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4740.070253] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4740.082860] Lustre: Skipped 8 previous similar messages [ 4740.109686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4740.122305] Lustre: Skipped 8 previous similar messages [ 4740.129381] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4740.137220] Lustre: Skipped 8 previous similar messages [ 4740.140990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4740.936243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4741.701268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4741.711804] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4741.724588] Lustre: Skipped 9 previous similar messages [ 4742.615516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4743.475703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4744.379010] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4745.267778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4746.322653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4747.356360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4748.185652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4748.934673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4749.720141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4750.583381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4751.366933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4752.147656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4753.034629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4753.887850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4754.563766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4755.401699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4756.172991] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4756.177622] Lustre: Skipped 18 previous similar messages [ 4756.202250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.227905] Lustre: Skipped 18 previous similar messages [ 4756.232908] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4756.237858] Lustre: Skipped 18 previous similar messages [ 4756.242936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4757.088562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4757.827259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4757.833106] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4757.840669] Lustre: Skipped 18 previous similar messages [ 4758.704738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.449298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4760.219935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4760.967499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4761.731610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4762.490788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4763.572699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4764.737281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4765.666264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.551193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4767.426605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4768.234863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.026043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.866303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.539043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.448086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4772.377701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.273554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4774.203721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.222403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.068486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4777.005658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4777.757489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.702263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4779.593596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.539622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4781.382223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.405232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.183894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4784.040157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4784.857550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.663532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4786.565247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4787.323469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.005134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.845909] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4788.854974] Lustre: Skipped 37 previous similar messages [ 4788.873380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.880344] Lustre: Skipped 37 previous similar messages [ 4788.883514] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4788.888173] Lustre: Skipped 37 previous similar messages [ 4788.891493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.717919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.616387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.632623] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4790.646276] Lustre: Skipped 37 previous similar messages [ 4791.378863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4792.457978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.511814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.517264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.479702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.376447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.197115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.247183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.324107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.159440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.216364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4802.218334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.085906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.994366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.823097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4805.601564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.396146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.297528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4808.215450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.021705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.859555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4810.694318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.689850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4812.574963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4813.577679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4814.486957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.440358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.301200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.239608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.077070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.937569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.897038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4820.752237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4821.888587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.735560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.611301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4824.481780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4825.399221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.373671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.193961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.031950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4829.006887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4829.665619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.387823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4831.217071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.094498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.902729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.705313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4834.564740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4835.452500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4836.403850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.429950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.275802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4839.199321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4840.040703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4841.121359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4842.159190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.038711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4844.097937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.229890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4846.310383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4847.589032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.642806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4849.596964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.785422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4851.634320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4852.706664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.646336] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4853.652666] Lustre: Skipped 69 previous similar messages [ 4853.694072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4853.707319] Lustre: Skipped 69 previous similar messages [ 4853.711652] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4853.722809] Lustre: Skipped 69 previous similar messages [ 4853.732206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4854.994132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.009715] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4855.025542] Lustre: Skipped 68 previous similar messages [ 4855.987969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4857.369743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4858.657308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.054272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4860.867761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4861.739808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4862.641075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4863.559715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4864.471914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4865.384469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4866.247291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.063198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4867.984035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4868.765594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4869.504201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4870.350498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4871.177389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4871.995395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4872.910438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4873.867858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4874.797866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4875.870614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4876.941232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4877.926469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4879.245948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4880.324953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4881.214896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4882.236444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4883.206082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4884.326844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.064128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4885.771163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4886.811367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4887.659976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4888.605883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4889.441521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4890.376422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4891.168608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4892.038354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4892.882260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4893.860775] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4893.870566] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4893.880576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4894.825389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4895.803294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4896.713773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4897.732992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4898.849721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4899.769867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4900.752952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4901.733242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4902.642700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4903.603947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4904.639893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4905.526242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4906.264121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4907.120643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4908.224981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4909.196532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4910.201539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4911.284352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4912.199990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4913.214803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4914.173399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4915.202259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4915.984913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4916.813324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4917.760957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.733607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4919.766063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4920.711611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4921.640617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4922.538351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.443609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4924.428805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4925.406471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.348548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.294768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4928.166753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4929.130268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.060413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.978735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.934745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4932.790255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4933.731463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4934.501822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4935.297344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4936.003197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4936.713807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4937.473720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4938.586973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.642723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4940.405369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.284304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.135055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.937034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.829518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.742530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.722023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4946.663887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.556662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.440909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.377671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.296504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.244464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4952.001306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4953.045060] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4953.066707] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4953.077928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.338385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.366546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.521635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.832634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.012313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.095876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4961.126679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.197338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4963.155540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.386033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.216135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.182856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.228127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.162408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.351521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.214493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.079356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.205854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.456562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.518948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.245104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.194862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.212904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.487392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.685504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.650905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.804034] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4981.816134] Lustre: Skipped 132 previous similar messages [ 4981.859264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4981.868478] Lustre: Skipped 132 previous similar messages [ 4981.872588] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4981.877974] Lustre: Skipped 132 previous similar messages [ 4981.881894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.993949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.010700] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4983.023717] Lustre: Skipped 132 previous similar messages [ 4983.911431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.882230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4985.977818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.939648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.067230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.301992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.418291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.605142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.765914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.737073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.746632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.773116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.887512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.800673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.891232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.954584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.899907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.947581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.063288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.091464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.531965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.964149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.328602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.386807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.332799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.689531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.779252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.839681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.054959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.413930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.921120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.453878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.885958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.160689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.612929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5024.433862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.066724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.817487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.171809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.599426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.485695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.459367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.334550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.419509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.284245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.532273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.676611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.606267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.718271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5039.144046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.338190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.395240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.521189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.653996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5044.532985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.651314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.489888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.598330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.524863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5049.669974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.557844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.699233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.723759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.703231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.826387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.892395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5056.949273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.112878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.012828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.051945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.964527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.817181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.028965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.998806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.005132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.939625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.957904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.857557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.044956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.191062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.393955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.124803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.027767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.188513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.353635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.456857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.871303] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5077.890838] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5077.904156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.815589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.027827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.138908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.788338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.841118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.822841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.973546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.184168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.044357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.107938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.253310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.319906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.282211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.275888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.567806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.675560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.710290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.776659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.762576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.846655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.679724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.813967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.592047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.641568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.499206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.356174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.181404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.050410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.897620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.898312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.690271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.530317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.195445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.105994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.914454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.014574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.859808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.605921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.390549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.400872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.411790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.342386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.279412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.261139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.300636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.183778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.063389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.719268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.488160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.218302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.022447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.705963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.534989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.283662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.001732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.713413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.695420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.421813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.409514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.186653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5133.918788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.790505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.762623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.601334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.558049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.426707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.313948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.256259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.384181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.308166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.539426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.448754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.510492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.321717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.126081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.905441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.682675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.522449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.456883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.179792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.074977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.830955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.573205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.374545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.197493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.154526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.087658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.799497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.619975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.516421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.505482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.367244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.371431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.251372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.101305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.987648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.797197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.625396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.574394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.559484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.332240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.126140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.910724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.830189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.022066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.010355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.109537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.871111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.586362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.355039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.203744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.061431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.944573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.161786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.041132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.802443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.650422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.707156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.770881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.596807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.533279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.502516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.341552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.144089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.026413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.866156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.666487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.548951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.332559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.120583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.843323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.588764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.367580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.270224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.952449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.855285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.701501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.531722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.281694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.267823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.264339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.478089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.223395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.050094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.911448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.804515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.663979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.566751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.483654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.506723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.372801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.228612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.186469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.167967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.154222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.939143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.773905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.780594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.655115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.507869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.359950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.522095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.504816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.480769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.327821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.393565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.130837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.000106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.716576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.604596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.279251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.114534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.012033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.794448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.570830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.561514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.324017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.210527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.018567] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5238.025412] Lustre: Skipped 266 previous similar messages [ 5238.046378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5238.070029] Lustre: Skipped 266 previous similar messages [ 5238.076431] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5238.088663] Lustre: Skipped 266 previous similar messages [ 5238.091410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.975286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.799964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.824165] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5239.834440] Lustre: Skipped 267 previous similar messages [ 5240.799875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.657505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.563155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.484127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.388208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.296362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.205895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.085762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.017173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.736241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.558505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.349347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.193062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.181498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.209168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.119802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.031290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.869812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.709353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.453524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.562643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.623804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.541941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.387152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.313631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.249891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.207453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.239955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.105263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.163361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.139482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.054330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.898585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.845734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.854485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.703378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.683150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.791917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.654745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.569609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.375727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.275497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.090813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.972279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.976384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.888932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.205582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.417746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.414387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.308861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.163473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.201152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.066959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.101636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.994803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.739706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.530302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.315388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.203528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.281844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.303449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.425829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.338649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.272335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.345290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.345672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.364907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.433579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.580469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.874898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.712801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.766483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.665025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.554917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.492910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.494774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.480819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.562283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.487200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.653591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.797508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.947787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.928837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.875678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.649710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.600866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.731528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.465608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.353364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.500280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.454377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.515560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.796501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.835878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.073130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.897036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.146832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.833375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.313206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.485453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.306447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.440844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.515989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.736784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.290167] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5343.297612] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5343.322383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.766879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.873883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.069488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.178570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.480497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.411110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.448933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.497127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.529943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.530496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.439727] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5355.455144] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5355.477235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.401322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.285506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.011638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.898545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.692902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.529520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.470592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.310619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.120269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.066201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.831036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.553293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.443726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.371663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.123056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.021935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.763298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.675019] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.492910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.440697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.361226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.268786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.155455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.892650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.774268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.593893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.537367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.444222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.285627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.221618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.107761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.968842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.854942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.726788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.522750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.421553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.172496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.161676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.143459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.069089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.027587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.082065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.154352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.228978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.264767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.194894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.244967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.277980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.559285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.691534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.949311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.254599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.565302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.444795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.389307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.231182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.152575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.021623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.839896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.741644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.671049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.701606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.993394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.761089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.689878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.580921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.508950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.428136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.390971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.109320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.997711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.767208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.512041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.312906] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5423.320281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.321125] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 5423.321155] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5423.321159] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 1 previous similar message [ 5424.020588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.816687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.732633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.818590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.728094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.985395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.942723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.842903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.765422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.741098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.849823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.689713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.835589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.650291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.449215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.260419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.162480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.881417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.513221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.128270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.992036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.824978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.653709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.612853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.679422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.756586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.626684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.523779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.281942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.173858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.139930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.111983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.993513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.853494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.754067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.542214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.405601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.247216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.105759] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5458.124618] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5458.127117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.054153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.071705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.989986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.137587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.274183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.160804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.216130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.051658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.876792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.836075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.715419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.548217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.566699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.469434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.401771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.495600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.518210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.744234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.705946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.542725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.484624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.497301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.636843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.793236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.651221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.565963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.524152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.485076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.330435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.192254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.163820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.141193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.102700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.088618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.917925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.065785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.839488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.691741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.575805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.516434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.307300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.171973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.968480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.727160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.558553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.414713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.272651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.167342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.053739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.999461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.845436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.781146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.639033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.471638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.391898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.356424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.220690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.380388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.230454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.019633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.812716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.592345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.379406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.261520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.170312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.200476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.040340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.002664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.835667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.729969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.270273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.212351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.244473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.173075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.235062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.408244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.365659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.423985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.425846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.346431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.208671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.053793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.904600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.721551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.693583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.543293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.606580] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5538.617467] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 5538.624305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.628870] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5538.628881] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 1 previous similar message [ 5539.466951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.439115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.372491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.249255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.153620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.125601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.958942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.783081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.738296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.618344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.423555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.421477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.210359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.175022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.066387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.146098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.942353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.848201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.859847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.791780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.547549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.493750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.393548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.401436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.389753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.670753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.895737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.240978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.444066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.616789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.776841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.983401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.030496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.205487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.317849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.829592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.066626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.039893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.531836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.893659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.177672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.209850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.329837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.836532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.877688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.992915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.016353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.186247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.121611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.121783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.094881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.067905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.071604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.090668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.952114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.231901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.161939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.219658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.539070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.842793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.926312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.436277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.771994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.123667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.581175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.926693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.931854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.489741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.898508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.168335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.537203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.983904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.119342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.357223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.254604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.025732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.789538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.325648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.250061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.171794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.416715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.826624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.840167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.664516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.406553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.757086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.996716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.351466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.659927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.019122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.171950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.280345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.380908] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5643.395065] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 5643.416545] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5643.419403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.426993] LustreError: 171894:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 1 previous similar message [ 5644.555394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.687683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.534255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.643426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.559619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.542612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.436701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.323917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.201522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.056710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.922214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.853482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.823863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.920421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.821529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.762426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.573987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.490327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.419469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.450378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.544825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.578227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.640549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.606688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.626481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.415162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.264693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.256814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.202168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.114986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.158353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.932682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.730040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.773554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.662667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.716962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.659838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.814989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.808958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.718835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.641626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.626957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.323888] LustreError: 173695:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6020.341795] LustreError: 171894:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6642.144739] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6642.163971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6642.172382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6642.184725] Lustre: Skipped 454 previous similar messages [ 6642.208318] Lustre: Skipped 1 previous similar message [ 6645.260241] Lustre: server umount lustre-OST0000 complete [ 6651.893535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6651.904163] LustreError: Skipped 1 previous similar message [ 6651.908131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6656.500369] Lustre: server umount lustre-MDT0000 complete [ 6657.505240] LustreError: 172602:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6657.521514] LustreError: 172602:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6660.725737] LustreError: 171856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789071917 with bad export cookie 7240856120895667213 [ 6660.735500] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6660.736190] LustreError: 171856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6661.027429] Lustre: server umount lustre-MDT0001 complete [ 6675.082274] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 6678.626096] Key type lgssc unregistered [ 6679.062811] LNet: 227127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6679.076765] LNetError: 227127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6679.111801] LNet: Removed LNI 192.168.206.127@tcp [ 6680.201235] Key type .llcrypt unregistered [ 6680.203300] Key type ._llcrypt unregistered [ 6710.879207] Key type ._llcrypt registered [ 6710.882449] Key type .llcrypt registered [ 6710.974599] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6712.521581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6712.684316] alg: No test for adler32 (adler32-zlib) [ 6714.056573] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 6714.368497] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 6716.075492] Key type lgssc registered [ 6718.002114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6728.358548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6728.434778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6730.436960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6736.112614] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6740.966402] LustreError: 228541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6740.985662] LustreError: 228541:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6745.254608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6745.696763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.487064] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6759.736702] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6765.507995] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6771.872784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6772.258075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6775.357046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6779.162665] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6786.151893] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6794.163080] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6795.911690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.926959] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6795.939881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6795.955396] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6795.956783] LustreError: 228561:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6798.170227] Lustre: lustre-OST0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 6800.872094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.874669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.888736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6800.911902] Lustre: Skipped 1 previous similar message [ 6800.918322] Lustre: Skipped 1 previous similar message [ 6803.260489] Lustre: lustre-OST0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 6804.332066] Lustre: server umount lustre-OST0000 complete [ 6816.023924] LustreError: 228561:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6840.658179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6841.345199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6850.271604] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6858.602841] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6860.296469] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6865.664360] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8bf44508c000.ost_server_uuid 50 [ 6865.831352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6865.851977] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6865.861973] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6865.868264] Lustre: Skipped 1 previous similar message [ 6867.350361] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8bf44508c000.ost_server_uuid in FULL state after 0 sec [ 6873.311354] Lustre: server umount lustre-OST0000 complete [ 6878.689050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6878.696605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.723171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6881.252861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.369950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6886.375915] Lustre: Skipped 3 previous similar messages [ 6887.391815] LustreError: 228561:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006e898a97 x1875977931147136/t0(0) o7->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 264/240 e 0 to 0 dl 0 ref 2 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6887.585900] Lustre: server umount lustre-MDT0000 complete [ 6891.488445] Lustre: 227852:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789072132/real 1789072132] req@000000009d1f3f58 x1875977931144064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789072148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6891.497130] LustreError: 229269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6891.498729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6891.555348] LustreError: 229269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6891.901991] LustreError: 228523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072148 with bad export cookie 304449074358716037 [ 6891.915884] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6891.920311] LustreError: 228523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6892.313925] Lustre: server umount lustre-MDT0001 complete [ 6907.599984] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 6912.682829] Key type lgssc unregistered [ 6913.012778] LNet: 233749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6913.023050] LNetError: 233749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6913.038342] LNet: Removed LNI 192.168.206.127@tcp [ 6913.931834] Key type .llcrypt unregistered [ 6913.947657] Key type ._llcrypt unregistered [ 6949.976930] Key type ._llcrypt registered [ 6949.978646] Key type .llcrypt registered [ 6950.095838] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6951.385275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6951.421351] alg: No test for adler32 (adler32-zlib) [ 6952.853510] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 6953.184137] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 6955.039196] Key type lgssc registered [ 6956.737306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6968.420161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6968.466405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.891109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6976.388933] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6981.100683] LustreError: 235478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6981.131307] LustreError: 235478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6984.988032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.437805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6989.900511] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6992.798308] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6999.437617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6999.913615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7006.967871] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7016.333918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7017.072815] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.326305] Lustre: lustre-OST0001: new disk, initializing [ 7017.330852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7017.339637] Lustre: 237787:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7017.526443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7023.287564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7023.302481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7023.508236] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7025.234632] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7037.723503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7047.424763] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7064.546396] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7064.555205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7064.562581] Lustre: Skipped 2 previous similar messages [ 7064.570849] Lustre: Skipped 2 previous similar messages [ 7069.462464] Lustre: server umount lustre-MDT0000 complete [ 7069.665631] LustreError: 235478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7069.681169] LustreError: 235478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7074.508881] LustreError: 235459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072331 with bad export cookie 1584079240964540283 [ 7074.511283] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7074.543190] LustreError: 235459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7074.786348] LustreError: 235474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.789569] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7074.836029] LustreError: 235474:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7074.847530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7074.875322] Lustre: Skipped 2 previous similar messages [ 7074.919981] Lustre: Skipped 2 previous similar messages [ 7075.421654] Lustre: server umount lustre-MDT0001 complete [ 7083.174315] Lustre: server umount lustre-OST0000 complete [ 7089.456322] Lustre: server umount lustre-OST0001 complete [ 7113.448575] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7125.605728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7126.468399] LustreError: 240895:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7126.598178] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.626234] LustreError: 240896:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7131.829095] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7141.861795] LustreError: 240895:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7141.878812] LustreError: 240895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7143.539244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7144.096916] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7149.232832] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7152.897759] Lustre: 242036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7159.481101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7166.893910] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7169.064938] LustreError: 242391:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7169.097722] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 7172.170301] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7176.556191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7176.855911] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7176.869895] Lustre: Skipped 1 previous similar message [ 7182.351509] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 7183.201272] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7191.180800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7194.471183] Lustre: 243908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7205.344388] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7205.349381] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7205.364425] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7207.904645] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7207.911441] Lustre: mylustre-MDT0000-lwp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7207.922958] Lustre: Skipped 2 previous similar messages [ 7207.971943] Lustre: Skipped 2 previous similar messages [ 7213.028605] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7213.042115] Lustre: Skipped 4 previous similar messages [ 7218.153989] Lustre: mylustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7218.324331] Lustre: server umount mylustre-MDT0000 complete [ 7221.692172] LustreError: 244108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072478 with bad export cookie 1584079240964543146 [ 7221.694654] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7221.712598] LustreError: 244108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7222.035044] Lustre: server umount mylustre-MDT0001 complete [ 7236.381072] Lustre: server umount mylustre-OST0000 complete [ 7238.367180] Lustre: 234742:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789072479/real 1789072479] req@ffff8b30704cea00 x1875978180789120/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789072495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7238.395707] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7241.342268] Lustre: server umount mylustre-OST0001 complete [ 7272.832635] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7285.358413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.959253] LustreError: 246221:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7285.975587] LustreError: 246221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7286.041568] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7291.387211] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7301.470035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7309.435335] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7312.830889] Lustre: 247366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7319.578721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7327.236000] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 7328.047314] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7331.337790] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7339.103488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7344.665570] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 7346.406672] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7356.057979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7360.829506] Lustre: 249239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7373.793201] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7373.805178] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7373.830504] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7373.833124] Lustre: Skipped 4 previous similar messages [ 7379.000965] Lustre: server umount tfs-MDT0000 complete [ 7380.456262] LustreError: 246221:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7380.493670] LustreError: 246221:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 7384.103386] LustreError: 246196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072640 with bad export cookie 1584079240964545323 [ 7384.104698] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7384.110116] LustreError: 246196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7384.431384] Lustre: server umount tfs-MDT0001 complete [ 7400.077412] Lustre: server umount tfs-OST0000 complete [ 7401.960343] Lustre: 234741:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789072642/real 1789072642] req@ffff8b30513a3480 x1875978180848000/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789072658 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7401.996285] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.014487] Lustre: Skipped 3 previous similar messages [ 7403.742832] Lustre: server umount tfs-OST0001 complete [ 7428.450470] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7444.035884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7445.134604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7445.145456] Lustre: Skipped 3 previous similar messages [ 7451.099293] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7460.227433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7466.033864] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7469.432905] Lustre: 252632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7476.628689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7486.219384] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 7486.480980] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7489.339495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7498.769368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7504.383752] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 7507.642970] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7518.543930] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7522.892801] Lustre: 254502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7531.324722] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 16:39:46 (1789072786) [ 7550.435439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.447331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7550.460035] Lustre: Skipped 3 previous similar messages [ 7550.484119] Lustre: Skipped 7 previous similar messages [ 7554.988762] Lustre: server umount lustre-MDT0000 complete [ 7555.559601] LustreError: 254551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7555.596834] LustreError: 254551:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 7559.039648] LustreError: 251465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789072815 with bad export cookie 1584079240964547241 [ 7559.051967] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7559.059197] LustreError: 251465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7565.974540] Lustre: server umount lustre-MDT0001 complete [ 7576.817111] Lustre: server umount lustre-OST0000 complete [ 7594.975141] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7595.274645] Lustre: server umount lustre-OST0001 complete [ 7604.270020] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 7622.180252] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7664.277172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.739295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7664.765105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7664.846272] Lustre: lustre-MDT0000: new disk, initializing [ 7664.938500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7664.943135] Lustre: Skipped 3 previous similar messages [ 7664.957473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7670.055183] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7681.924217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.201895] Lustre: 257715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7682.330312] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7682.343099] Lustre: Skipped 1 previous similar message [ 7682.517966] Lustre: lustre-MDT0001: new disk, initializing [ 7682.681725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7682.699119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7689.358171] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7703.690435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7703.970144] Lustre: lustre-OST0000: new disk, initializing [ 7703.973981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7703.980291] Lustre: 259298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7705.825452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7705.835697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7706.059928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7711.704193] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7725.516824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7726.057096] Lustre: lustre-OST0001: new disk, initializing [ 7726.066878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7726.100483] Lustre: 260172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7727.414825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7727.439875] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7727.680615] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7739.431974] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7758.062980] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 16:43:31 (1789073011) [ 7774.265532] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 16:43:48 (1789073028) [ 7794.186712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.189298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.210209] Lustre: Skipped 3 previous similar messages [ 7794.219591] Lustre: Skipped 4 previous similar messages [ 7799.549140] Lustre: server umount lustre-OST0000 complete [ 7810.810383] Lustre: server umount lustre-MDT0000 complete [ 7814.627705] LustreError: 257723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7814.652143] LustreError: 257723:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7815.281967] LustreError: 257706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073071 with bad export cookie 1584079240964549019 [ 7815.286574] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7815.311358] LustreError: 257706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7821.689681] Lustre: server umount lustre-MDT0001 complete [ 7833.297673] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 7907.773815] Lustre: server umount lustre-OST0001 complete [ 7915.437595] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 7935.062502] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7981.691775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.193882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7982.266114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7982.455398] Lustre: lustre-MDT0000: new disk, initializing [ 7982.613818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7982.633462] Lustre: Skipped 3 previous similar messages [ 7982.685412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7990.530374] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8004.540261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.700945] Lustre: 265032:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8004.758755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8004.761816] Lustre: Skipped 1 previous similar message [ 8004.825950] Lustre: lustre-MDT0001: new disk, initializing [ 8004.993339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8005.009430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8010.502192] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8023.294930] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8032.049263] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8042.275351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8042.851886] Lustre: lustre-OST0000: new disk, initializing [ 8042.861896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8042.869728] Lustre: 267003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8044.042175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8044.060238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8044.252053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8052.542885] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8064.053676] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8070.118200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8070.128455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8070.145670] Lustre: Skipped 4 previous similar messages [ 8070.151522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8070.156798] Lustre: Skipped 9 previous similar messages [ 8076.054067] Lustre: server umount lustre-OST0000 complete [ 8081.889324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8086.712129] Lustre: server umount lustre-MDT0000 complete [ 8092.936556] LustreError: 265025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073349 with bad export cookie 1584079240964550489 [ 8092.944636] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8092.952691] LustreError: 265025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8093.602797] Lustre: server umount lustre-MDT0001 complete [ 8109.124401] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 8112.496831] Key type lgssc unregistered [ 8113.049098] LNet: 268765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8113.065366] LNetError: 268765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8113.114363] LNet: Removed LNI 192.168.206.127@tcp [ 8113.991349] Key type .llcrypt unregistered [ 8113.993809] Key type ._llcrypt unregistered [ 8145.343586] Key type ._llcrypt registered [ 8145.346486] Key type .llcrypt registered [ 8145.488167] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8146.750933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8146.981246] alg: No test for adler32 (adler32-zlib) [ 8147.990344] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 8148.287429] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 8149.967333] Key type lgssc registered [ 8151.192573] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8159.926548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8159.951180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.343902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8166.264054] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8171.492362] LustreError: 270223:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8171.531406] LustreError: 270223:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8175.521558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.055985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8181.593606] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8189.875416] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8195.762721] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8204.483971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8205.082376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8207.095325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8212.695727] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8221.523397] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8222.513761] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8225.763806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8225.767233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8225.783984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8225.796947] Lustre: Skipped 1 previous similar message [ 8225.804048] Lustre: Skipped 1 previous similar message [ 8226.943935] Lustre: server umount lustre-OST0000 complete [ 8230.881136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8230.889231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8230.892915] Lustre: Skipped 1 previous similar message [ 8236.003234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8241.124600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8241.142879] Lustre: Skipped 2 previous similar messages [ 8245.215129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8245.411096] Lustre: server umount lustre-MDT0000 complete [ 8246.243736] LustreError: 271420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.264372] LustreError: 271420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8248.918355] LustreError: 270203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073505 with bad export cookie 4233353890767939177 [ 8248.924371] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8248.932530] LustreError: 270203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8249.418390] Lustre: server umount lustre-MDT0001 complete [ 8260.970908] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 8264.232987] Key type lgssc unregistered [ 8264.549457] LNet: 273808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8264.553667] LNetError: 273808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8264.576691] LNet: Removed LNI 192.168.206.127@tcp [ 8265.585549] Key type .llcrypt unregistered [ 8265.590862] Key type ._llcrypt unregistered [ 8281.989385] Key type ._llcrypt registered [ 8281.991253] Key type .llcrypt registered [ 8282.097569] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8283.123252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8283.221127] alg: No test for adler32 (adler32-zlib) [ 8284.396296] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 8284.716605] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 8286.399213] Key type lgssc registered [ 8287.623913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8296.408301] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8296.424819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8311.967589] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8317.087405] LustreError: 274998:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.127@tcp: failed processing log, type 4: rc = -110 [ 8343.711379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8351.838627] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8362.059080] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8364.725412] Lustre: server umount lustre-OST0000 complete [ 8380.279229] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 8383.284334] Key type lgssc unregistered [ 8383.601046] LNet: 276629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8383.610708] LNetError: 276629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8383.629832] LNet: Removed LNI 192.168.206.127@tcp [ 8384.850553] Key type .llcrypt unregistered [ 8384.858073] Key type ._llcrypt unregistered [ 8433.775552] Key type ._llcrypt registered [ 8433.779327] Key type .llcrypt registered [ 8433.923281] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8434.932682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8434.966327] alg: No test for adler32 (adler32-zlib) [ 8436.005640] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 8436.338423] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 8437.991210] Key type lgssc registered [ 8439.497675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8453.584409] loop0: detected capacity change from 0 to 204800000 [ 8453.741504] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8453.751724] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8454.338898] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8455.524606] loop0: detected capacity change from 0 to 204800000 [ 8455.743377] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8455.760514] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8456.291441] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8457.329678] loop0: detected capacity change from 0 to 204800000 [ 8457.575125] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8457.584281] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8457.734201] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8458.957409] loop0: detected capacity change from 0 to 204800000 [ 8459.115598] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8459.134255] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8459.328996] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8460.429882] loop0: detected capacity change from 0 to 204800000 [ 8460.464643] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8464.618860] loop0: detected capacity change from 0 to 204800000 [ 8464.649864] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8468.617912] loop0: detected capacity change from 0 to 204800000 [ 8468.644696] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8473.446652] loop0: detected capacity change from 0 to 204800000 [ 8473.472135] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8477.859319] loop0: detected capacity change from 0 to 204800000 [ 8477.970292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8478.036870] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8478.134796] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8478.141236] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8478.150657] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8478.162840] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8478.173414] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8478.184735] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8478.193816] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8478.201939] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8478.207815] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8478.217886] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8478.227276] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8478.236377] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8478.246770] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8478.258114] Lustre: 280042:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8485.761390] Lustre: server umount MGS complete [ 8486.786138] loop0: detected capacity change from 0 to 204800000 [ 8486.888640] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8487.636204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8487.687547] Lustre: lustre-MDT0000: Aborting client recovery [ 8488.625521] loop1: detected capacity change from 0 to 204800000 [ 8488.772975] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.924732] Lustre: 280475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8488.938475] Lustre: 280475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8488.966389] Lustre: 280475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8488.990877] Lustre: 280475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8489.011638] Lustre: 280475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8489.030975] Lustre: 280475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8489.056532] Lustre: 280475:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8489.095783] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8489.415075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8489.459306] Lustre: lustre-MDT0001: Aborting client recovery [ 8490.632397] loop2: detected capacity change from 0 to 204800000 [ 8490.746200] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8490.864695] Lustre: 280554:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8490.888716] Lustre: 280554:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8490.910828] Lustre: 280554:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8490.948583] Lustre: 280554:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8490.983978] Lustre: 280554:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8491.013353] Lustre: 280554:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8491.052703] Lustre: 280554:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8491.232465] Lustre: Found index 0 for lustre-OST0000, updating log [ 8491.287390] Lustre: 280571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8491.363201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8491.392329] Lustre: lustre-OST0000: Aborting recovery [ 8492.471020] loop3: detected capacity change from 0 to 204800000 [ 8492.612318] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8492.796959] Lustre: 280636:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8492.813578] Lustre: 280636:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8492.822611] Lustre: 280636:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8492.831670] Lustre: 280636:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8492.840153] Lustre: 280636:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8492.849452] Lustre: 280636:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8492.858919] Lustre: 280636:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8492.878624] Lustre: Found index 1 for lustre-OST0001, updating log [ 8492.902201] Lustre: 280642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8492.978133] Lustre: lustre-OST0001: Aborting recovery [ 8498.081653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8498.089586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8498.137461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8498.157789] LustreError: 280419:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8498.204687] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8500.141257] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8500.142260] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8501.726677] LustreError: 280386:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 2038us: rc = -2 [ 8502.234312] LustreError: 280387:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 620us: rc = -2 [ 8502.250262] LustreError: 280387:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 17 previous similar messages [ 8520.159935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8520.179170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8520.236758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8525.793043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8525.808504] Lustre: Skipped 6 previous similar messages [ 8530.400695] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8530.912270] LustreError: 280385:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8530.935592] LustreError: 280385:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 8531.016553] Lustre: server umount lustre-MDT0000 complete [ 8532.401462] LustreError: 280369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789073789 with bad export cookie 8781896167023412053 [ 8532.426218] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8536.036295] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.047172] LustreError: 280384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8536.053493] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8536.062719] Lustre: Skipped 4 previous similar messages [ 8536.108501] LustreError: 280384:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8537.056271] LustreError: 280383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.101818] LustreError: 280383:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8539.119537] Lustre: server umount lustre-MDT0001 complete [ 8546.334659] Lustre: server umount lustre-OST0000 complete [ 8554.265359] Lustre: server umount lustre-OST0001 complete [ 8565.702596] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 16:56:59 (1789073819) [ 8595.259755] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8642.229262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.692766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8642.761925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8642.922335] Lustre: lustre-MDT0000: new disk, initializing [ 8643.076528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8643.089244] Lustre: Skipped 1 previous similar message [ 8643.137631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8649.867355] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8657.747904] Lustre: server umount lustre-MDT0000 complete [ 8678.557219] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 16:58:53 (1789073933) [ 8710.329107] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 8722.622248] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8765.648126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8766.246837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8766.334115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8766.343434] Lustre: Skipped 1 previous similar message [ 8766.502594] Lustre: lustre-MDT0000: new disk, initializing [ 8766.723402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.764818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8771.819400] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8789.530726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.729576] Lustre: 287754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8789.819503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8789.824756] Lustre: Skipped 1 previous similar message [ 8789.895829] Lustre: lustre-MDT0001: new disk, initializing [ 8790.006198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8790.076898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8790.105279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8796.242861] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8807.018263] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8813.917713] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8822.163931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8822.565431] Lustre: lustre-OST0000: new disk, initializing [ 8822.571792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8822.596713] Lustre: 289734:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8822.709792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8829.867844] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8831.085365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8831.112497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8831.253924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8840.160942] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8845.178228] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8856.005643] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8866.255067] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8913.376208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8913.389351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.399394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8913.404447] Lustre: Skipped 1 previous similar message [ 8913.437459] Lustre: Skipped 4 previous similar messages [ 8918.504221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8918.517640] Lustre: Skipped 1 previous similar message [ 8919.278897] Lustre: server umount lustre-OST0000 complete [ 8923.619206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.225073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8928.235157] LustreError: Skipped 1 previous similar message [ 8928.239511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.291120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8928.305574] Lustre: Skipped 1 previous similar message [ 8930.159401] Lustre: server umount lustre-MDT0000 complete [ 8933.856146] LustreError: 287764:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8933.888686] LustreError: 287764:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8934.483565] LustreError: 287745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074191 with bad export cookie 8781896167023415014 [ 8934.506825] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8934.511349] LustreError: 287745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8934.905555] Lustre: server umount lustre-MDT0001 complete [ 8944.413178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8949.118084] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8953.246850] Lustre: server umount MGS complete [ 8966.009307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8966.879238] LustreError: 293696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8966.940577] LustreError: 293709:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8967.057908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8973.040550] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8976.361123] LustreError: 293696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8976.381354] LustreError: 293696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8983.712042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.422336] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9003.711247] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9013.686381] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9021.802620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9030.234760] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9038.931215] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9070.568471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9070.580542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9070.605329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9070.611202] Lustre: Skipped 1 previous similar message [ 9074.908664] Lustre: server umount lustre-OST0000 complete [ 9081.312498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9081.315685] LustreError: Skipped 1 previous similar message [ 9085.368488] Lustre: server umount lustre-MDT0000 complete [ 9087.978910] LustreError: 293697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9088.009514] LustreError: 293697:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9089.402586] LustreError: 296424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074346 with bad export cookie 8781896167023416869 [ 9089.408531] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9089.426669] LustreError: 296424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9089.758842] Lustre: server umount lustre-MDT0001 complete [ 9102.366667] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 9105.200169] Key type lgssc unregistered [ 9105.480756] LNet: 297641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9105.492721] LNetError: 297641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9105.515900] LNet: Removed LNI 192.168.206.127@tcp [ 9106.621108] Key type .llcrypt unregistered [ 9106.624118] Key type ._llcrypt unregistered [ 9143.987283] Key type ._llcrypt registered [ 9143.991523] Key type .llcrypt registered [ 9144.090340] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 9158.680864] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9159.689156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9159.735446] alg: No test for adler32 (adler32-zlib) [ 9160.796228] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 9161.040812] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 9162.703240] Key type lgssc registered [ 9163.685670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9199.807582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9199.836379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9201.119432] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9201.168214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9201.293804] Lustre: lustre-MDT0000: new disk, initializing [ 9201.494796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.509977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9206.344902] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9216.686197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9216.883967] Lustre: 301596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9216.957846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9216.966724] Lustre: Skipped 1 previous similar message [ 9217.095417] Lustre: lustre-MDT0001: new disk, initializing [ 9217.212875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9217.251901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9217.264419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9221.514311] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9230.663994] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9235.849453] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9243.033600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9243.341900] Lustre: lustre-OST0000: new disk, initializing [ 9243.347506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9243.352960] Lustre: 303566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9243.454157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9248.306488] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9252.397223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9252.405900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9252.517086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9256.637471] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9264.238322] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9275.398983] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9281.215047] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9289.260570] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9329.124669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9329.132467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9329.150321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9329.642835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9329.656106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9329.676153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9334.634591] Lustre: server umount lustre-OST0000 complete [ 9339.873766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.876682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9339.905060] Lustre: Skipped 1 previous similar message [ 9345.002223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9345.010310] Lustre: Skipped 2 previous similar messages [ 9345.309176] Lustre: server umount lustre-MDT0000 complete [ 9349.122526] LustreError: 305017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074605 with bad export cookie 4807446844847040371 [ 9349.124134] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9349.140240] LustreError: 305017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9349.527822] Lustre: server umount lustre-MDT0001 complete [ 9358.211833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.733049] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9367.677442] Lustre: server umount MGS complete [ 9379.455452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.884972] LustreError: 307445:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9379.977088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9384.306596] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9385.443223] LustreError: 307432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9385.465411] LustreError: 307432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9389.542554] LustreError: 307433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9393.152715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9393.549142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9397.673612] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9405.372757] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9410.089767] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9416.099508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9416.509927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9421.554219] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9428.773829] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9455.584238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9455.589464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9455.596625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9455.629038] Lustre: Skipped 1 previous similar message [ 9456.907230] Lustre: server umount lustre-OST0000 complete [ 9460.708503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9465.312379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9465.319668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9465.325216] Lustre: Skipped 2 previous similar messages [ 9466.649880] Lustre: server umount lustre-MDT0000 complete [ 9470.129178] LustreError: 307412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789074726 with bad export cookie 4807446844847042212 [ 9470.135869] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9470.142234] LustreError: 307412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9470.414645] Lustre: server umount lustre-MDT0001 complete [ 9479.835160] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 9482.091699] Key type lgssc unregistered [ 9482.329478] LNet: 311376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9482.336797] LNetError: 311376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9482.354025] LNet: Removed LNI 192.168.206.127@tcp [ 9483.191109] Key type .llcrypt unregistered [ 9483.193264] Key type ._llcrypt unregistered [ 9501.814839] Key type ._llcrypt registered [ 9501.817320] Key type .llcrypt registered [ 9501.912409] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 9513.916221] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9514.789872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9514.962049] alg: No test for adler32 (adler32-zlib) [ 9516.076590] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [ 9516.272789] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 9518.000520] Key type lgssc registered [ 9519.311059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9547.877063] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9573.403225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9573.498586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9574.974513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9575.053482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9575.164477] Lustre: lustre-MDT0000: new disk, initializing [ 9575.251882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9575.275104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9580.151305] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9590.427318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.882367] Lustre: lustre-OST0000: new disk, initializing [ 9590.887931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9590.891602] Lustre: Skipped 1 previous similar message [ 9590.895850] Lustre: 315403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9591.003921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9591.545767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9591.556586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9591.597550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9597.158220] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10426.364477] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10426.387842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10428.040874] Lustre: server umount lustre-MDT0000 complete [10432.806072] LustreError: 314589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075689 with bad export cookie 2775792210332446305 [10432.830321] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10432.837889] LustreError: 314589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10433.141134] Lustre: server umount lustre-OST0000 complete [10465.105542] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [10480.788476] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [10531.628316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10531.960336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10532.058806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10532.251617] Lustre: lustre-MDT0000: new disk, initializing [10532.504917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10532.597535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10539.116452] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10551.482961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.618663] Lustre: 319760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10551.688378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10551.692882] Lustre: Skipped 1 previous similar message [10551.761717] Lustre: lustre-MDT0001: new disk, initializing [10551.859700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10551.915277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10551.933371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10558.413532] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10569.351341] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10576.548741] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10586.657734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10587.305922] Lustre: lustre-OST0000: new disk, initializing [10587.313534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10587.325379] Lustre: 321728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10587.511949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10596.466605] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10597.082502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10597.104865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10597.453085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10608.213760] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10614.361337] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10614.563210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10620.125033] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10620.590832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10622.946857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10622.950959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10622.983186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10622.999550] Lustre: Skipped 1 previous similar message [10628.064278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10628.084130] Lustre: Skipped 1 previous similar message [10633.189665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10633.198825] Lustre: Skipped 1 previous similar message [10637.283857] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10638.175323] Lustre: server umount lustre-OST0000 complete [10648.548710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10648.555561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10648.591190] Lustre: Skipped 2 previous similar messages [10648.633510] Lustre: Skipped 1 previous similar message [10650.705767] Lustre: server umount lustre-MDT0000 complete [10654.617488] LustreError: 319751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789075911 with bad export cookie 2775792210335387124 [10654.626705] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10654.628412] LustreError: 319751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10655.216468] Lustre: server umount lustre-MDT0001 complete [10659.138483] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10661.240992] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 17:31:55 (1789075915) [10668.815546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10669.532484] LustreError: 324043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10669.548277] LustreError: 324043:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10669.636692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10674.658339] LustreError: 324044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10675.106685] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10679.780619] LustreError: 324043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.898377] LustreError: 324044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10686.627127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10687.370376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10693.361847] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10702.638713] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10708.817470] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10716.990090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10717.577855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10724.644723] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10733.995984] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10742.711874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10743.016710] Lustre: lustre-OST0001: new disk, initializing [10743.021874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10743.033501] Lustre: 326883:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10743.175710] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10749.118374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10749.132107] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10749.216185] LustreError: 327164:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10750.242065] LustreError: 327164:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10750.276105] LustreError: 327164:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10751.251603] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10751.330846] LustreError: 327176:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10751.346164] LustreError: 327176:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10753.384191] LustreError: 327164:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10753.407685] LustreError: 327164:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10757.551596] LustreError: 327176:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10757.567332] LustreError: 327176:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [10763.665050] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10764.098772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10765.797195] LustreError: 327176:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10765.823046] LustreError: 327176:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [10772.783180] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8bf44508f800.ost_server_uuid 50 [10774.493834] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8bf44508f800.ost_server_uuid in FULL state after 0 sec [10781.026605] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bf44508f800.ost_server_uuid 50 [10782.498797] LustreError: 327164:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10782.511439] LustreError: 327164:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 31 previous similar messages [10782.991166] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8bf44508f800.ost_server_uuid in IDLE state after 0 sec [10785.736465] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10800.622242] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10800.631524] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10800.651064] Lustre: Skipped 1 previous similar message [10805.740513] LustreError: 325867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10805.772918] LustreError: 325867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10806.072297] Lustre: server umount lustre-OST0001 complete [10813.922173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10813.939183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10815.983652] LustreError: 325867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10816.006128] LustreError: 325867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10818.867946] Lustre: server umount lustre-OST0000 complete [10825.696976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10825.704243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.729355] Lustre: Skipped 1 previous similar message [10825.741774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.756405] Lustre: Skipped 5 previous similar messages [10830.394799] Lustre: server umount lustre-MDT0000 complete [10835.989807] LustreError: 324026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076092 with bad export cookie 2775792210335388006 [10835.997343] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10836.012386] LustreError: 324026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10837.031151] Lustre: server umount lustre-MDT0001 complete [10851.728877] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [10854.728934] Key type lgssc unregistered [10854.969313] LNet: 329574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10854.977871] LNetError: 329574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10855.002472] LNet: Removed LNI 192.168.206.127@tcp [10856.136071] Key type .llcrypt unregistered [10856.142472] Key type ._llcrypt unregistered [10888.877320] Key type ._llcrypt registered [10888.881683] Key type .llcrypt registered [10889.111631] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [10891.063083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10891.096302] alg: No test for adler32 (adler32-zlib) [10892.387104] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [10892.613887] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [10894.279630] Key type lgssc registered [10895.863641] Lustre: Echo OBD driver; http://www.lustre.org/ [10905.027790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10905.076139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10906.839489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10912.263411] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10916.426092] LustreError: 330992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10916.472698] LustreError: 330992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10921.447728] LustreError: 330991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10922.052636] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10926.567057] LustreError: 330992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10931.556405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10931.689905] LustreError: 330991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.518037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10939.160633] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10950.270871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10950.913369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10951.924028] LustreError: 332671:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.943244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10958.447074] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10963.452838] LustreError: 332672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.468897] LustreError: 332672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10966.693988] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10975.128646] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10975.645442] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10980.867322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10986.496609] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10998.474813] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11007.556480] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11007.865955] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11119.072603] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11119.074274] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11119.084063] LustreError: Skipped 1 previous similar message [11119.085713] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11119.122404] Lustre: Skipped 1 previous similar message [11122.153913] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11122.160548] Lustre: Skipped 2 previous similar messages [11124.722559] Lustre: server umount lustre-OST0001 complete [11127.268926] LustreError: 332674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.283808] LustreError: 332674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11136.492833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11136.508385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11136.528969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11136.539643] Lustre: Skipped 1 previous similar message [11142.629369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11142.640920] Lustre: Skipped 3 previous similar messages [11147.231129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11147.503595] Lustre: server umount lustre-OST0000 complete [11152.356374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11152.368227] LustreError: Skipped 1 previous similar message [11152.376982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.407787] Lustre: Skipped 1 previous similar message [11152.420057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11158.541361] Lustre: server umount lustre-MDT0000 complete [11163.110246] LustreError: 332562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.120985] LustreError: 332562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [11163.256145] LustreError: 330971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076419 with bad export cookie 11615899737401083067 [11163.264335] LustreError: 330971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11163.265585] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11164.125220] Lustre: server umount lustre-MDT0001 complete [11178.431753] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11183.898801] Key type lgssc unregistered [11184.328627] LNet: 335939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11184.347118] LNetError: 335939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11184.383330] LNet: Removed LNI 192.168.206.127@tcp [11186.080590] Key type .llcrypt unregistered [11186.083966] Key type ._llcrypt unregistered [11241.267737] Key type ._llcrypt registered [11241.269852] Key type .llcrypt registered [11241.419132] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11243.621577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11244.268071] alg: No test for adler32 (adler32-zlib) [11245.770707] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [11246.297551] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [11248.175155] Key type lgssc registered [11250.427123] Lustre: Echo OBD driver; http://www.lustre.org/ [11264.806039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11264.839266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11266.520103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11271.127532] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11276.768794] LustreError: 338235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11276.795961] LustreError: 338235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11280.104557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11280.652882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11284.542715] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11287.675707] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11295.425299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11295.989170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11303.780599] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11306.289352] LustreError: 339726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.431207] LustreError: 339726:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.461268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11311.468145] LustreError: 339726:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11315.246096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11315.579412] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11320.845789] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11322.249640] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11332.190646] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11341.533069] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11356.653469] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11356.661451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11356.684630] Lustre: Skipped 3 previous similar messages [11356.687745] Lustre: Skipped 2 previous similar messages [11361.766170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11361.772051] Lustre: Skipped 2 previous similar messages [11362.139485] Lustre: server umount lustre-MDT0000 complete [11365.549487] LustreError: 338214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076622 with bad export cookie 15125772635577924804 [11365.557377] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11365.569777] LustreError: 338214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11365.972757] Lustre: server umount lustre-MDT0001 complete [11379.877876] Lustre: server umount lustre-OST0000 complete [11382.751191] Lustre: 337496:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789076623/real 1789076623] req@ffff8b2f4f3c4700 x1875982682227456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789076639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11382.792346] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.431229] Lustre: server umount lustre-OST0001 complete [11402.538985] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11413.532889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11413.860496] LustreError: 343502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11413.919280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11418.880302] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11419.109186] LustreError: 343503:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11429.353734] LustreError: 343502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11429.366147] LustreError: 343502:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11431.121350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11431.960547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11440.131790] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11444.743875] Lustre: 344645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11452.818061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11453.261386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11456.374420] LustreError: 345000:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11458.430584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11459.249779] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11469.541081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11475.559301] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11477.811077] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11489.269503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11494.283025] Lustre: 346516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11501.036360] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11501.039990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11501.065169] Lustre: Skipped 2 previous similar messages [11501.076621] Lustre: Skipped 1 previous similar message [11505.433893] Lustre: server umount lustre-MDT0000 complete [11506.146434] LustreError: 343502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.165767] LustreError: 343502:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [11509.711924] LustreError: 343484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789076766 with bad export cookie 15125772635577926505 [11509.722575] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11509.732215] LustreError: 343484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11510.104251] Lustre: server umount lustre-MDT0001 complete [11524.227851] Lustre: server umount lustre-OST0000 complete [11527.135334] Lustre: 337495:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789076767/real 1789076767] req@ffff8b2f438cdc00 x1875982682284160/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789076783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11527.189642] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11527.222907] Lustre: Skipped 1 previous similar message [11528.518477] Lustre: server umount lustre-OST0001 complete [11533.513935] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11535.194139] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11536.930624] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 17:46:31 (1789076791) [11561.335606] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11570.534039] loop0: detected capacity change from 0 to 1099511627776 [11571.305505] SGI XFS with ACLs, security attributes, quota, no debug enabled [11571.310839] *** VALIDATE xfs *** [11571.330964] XFS (loop0): Mounting V5 Filesystem [11571.405237] XFS (loop0): Ending clean mount [11572.783466] loop1: detected capacity change from 0 to 18691697672192 [11699.235454] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [11709.889600] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [11745.798355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11746.088771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11746.116486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11746.206341] Lustre: lustre-MDT0000: new disk, initializing [11746.329083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11746.342839] Lustre: Skipped 1 previous similar message [11746.364926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11751.197674] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11762.745225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11762.851615] Lustre: 351654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11762.908685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11762.911990] Lustre: Skipped 1 previous similar message [11762.981657] Lustre: lustre-MDT0001: new disk, initializing [11763.151102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11763.166674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11767.833946] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11776.709441] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11782.132490] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11789.270904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11789.609827] Lustre: lustre-OST0000: new disk, initializing [11789.613598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11789.622929] Lustre: 353622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11791.649420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11791.665744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11791.810637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11796.174056] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11806.063116] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11811.368375] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11811.612671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11816.509449] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11816.842829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11822.564030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11822.580029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11822.589113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11822.597134] Lustre: Skipped 3 previous similar messages [11827.691795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11827.705732] Lustre: Skipped 1 previous similar message [11833.312468] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11833.722613] Lustre: server umount lustre-OST0000 complete [11839.982654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11839.995370] LustreError: Skipped 1 previous similar message [11840.001232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11840.021587] Lustre: Skipped 1 previous similar message [11840.027127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11840.033131] Lustre: Skipped 2 previous similar messages [11844.660800] Lustre: server umount lustre-MDT0000 complete [11848.163468] LustreError: 351664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11848.193258] LustreError: 351664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11848.349653] LustreError: 351644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077105 with bad export cookie 15125772635577928059 [11848.350974] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11848.380257] LustreError: 351644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11848.713706] Lustre: server umount lustre-MDT0001 complete [11858.420821] XFS (loop0): Unmounting Filesystem [11864.752308] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 17:51:58 (1789077118) [11873.807709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11874.329453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11874.338569] Lustre: Skipped 2 previous similar messages [11880.056868] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11891.254271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11899.497089] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11909.355338] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11916.543176] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11923.518179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11931.304436] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [11940.045263] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11957.735818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11957.755844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11957.779291] Lustre: Skipped 1 previous similar message [11957.811405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11957.823472] Lustre: Skipped 2 previous similar messages [11961.600636] Lustre: server umount lustre-OST0000 complete [11968.500804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11973.849447] Lustre: server umount lustre-MDT0000 complete [11979.234763] LustreError: 356184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11979.244311] LustreError: 356184:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [11979.542876] LustreError: 358031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789077236 with bad export cookie 15125772635577928941 [11979.557733] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11979.576059] LustreError: 358031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11980.000759] Lustre: server umount lustre-MDT0001 complete [11994.853589] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [11998.209654] Key type lgssc unregistered [11998.675311] LNet: 359815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11998.683056] LNetError: 359815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11998.721693] LNet: Removed LNI 192.168.206.127@tcp [11999.748144] Key type .llcrypt unregistered [11999.751366] Key type ._llcrypt unregistered [12033.568144] Key type ._llcrypt registered [12033.569831] Key type .llcrypt registered [12033.950376] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [12037.820552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12037.889836] alg: No test for adler32 (adler32-zlib) [12038.875423] Lustre: Lustre: Build Version: 2.17.58_40_g706ff57 [12039.077225] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [12040.743230] Key type lgssc registered [12043.531930] Lustre: Echo OBD driver; http://www.lustre.org/ [12054.944071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12054.956049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12056.448761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12061.663383] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12066.789459] LustreError: 361384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12066.806358] LustreError: 361384:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12070.046938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12070.443193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12074.377984] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12080.967217] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12086.295996] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12092.089527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12092.515576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12099.270883] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12108.113509] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12116.448591] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12116.457257] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12116.478285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12117.476138] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12117.483100] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12117.524450] Lustre: Skipped 1 previous similar message [12120.400703] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [12122.535499] Lustre: server umount lustre-MDT0001 complete [12122.594945] LustreError: 362637:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.649667] LustreError: 362637:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12125.515456] LustreError: 361378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.738793] LustreError: 362113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.758247] LustreError: 362113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12132.840072] LustreError: 361384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12132.863524] LustreError: 361384:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12134.102904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12134.266774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12134.270842] Lustre: Found index 1 for lustre-MDT0001, updating log [12134.323654] Lustre: 361371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12134.935829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12140.026315] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12141.984466] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12143.138579] LustreError: 364866:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12143.155182] LustreError: 364866:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12143.184720] LustreError: 361376:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12143.329050] LustreError: 364880:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.127@tcp: cfg command failed: rc = -17 [12143.348097] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12143.348097] [12156.253842] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [12160.486676] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12160.502378] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12160.525699] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12160.573833] Lustre: Skipped 1 previous similar message [12162.832466] Lustre: server umount lustre-MDT0001 complete [12165.601470] LustreError: 361383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12165.629928] LustreError: 361383:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12177.707741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12177.837554] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12177.845872] Lustre: Found index 1 for lustre-MDT0001, updating log [12177.944659] Lustre: 361372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12178.414594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12183.533256] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12183.587535] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12188.157855] LustreError: 366060:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12188.166394] LustreError: 366060:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12188.185545] LustreError: 361376:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12188.189848] LustreError: 361376:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12188.296233] LustreError: 366063:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12188.309523] LustreError: 366063:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12188.319156] LustreError: 366063:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.127@tcp: cfg command failed: rc = -17 [12188.336067] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.206.127@tcp [12188.336067] [12490.733658] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12490.734971] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12490.757145] Lustre: Skipped 1 previous similar message [12490.778944] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12490.801984] Lustre: Skipped 2 previous similar messages [12495.596841] Lustre: server umount lustre-MDT0001 complete [12495.857355] LustreError: 362113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.893220] LustreError: 362113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [12509.400890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12509.597775] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12509.609953] Lustre: Found index 1 for lustre-MDT0001, updating log [12509.642193] Lustre: 361372:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12509.896245] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12515.302598] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12516.120120] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12519.944157] LustreError: 373993:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12519.958017] LustreError: 373993:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12519.967973] LustreError: 361376:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12519.977281] LustreError: 361376:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12520.055952] LustreError: 373996:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12520.074749] LustreError: 373996:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.127@tcp: cfg command failed: rc = -114 [12520.084674] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12520.084674] [12842.986119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12842.986457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12842.989361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12842.989374] Lustre: Skipped 1 previous similar message [12843.000212] Lustre: Skipped 1 previous similar message [12845.107505] Lustre: server umount lustre-MDT0000 complete [12848.098045] LustreError: 362113:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12848.143085] LustreError: 362113:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [12849.005039] LustreError: 363226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078105 with bad export cookie 17876531005951561615 [12849.010658] LustreError: 363226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12849.016776] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12853.220786] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.004020] Lustre: server umount lustre-MDT0001 complete [12861.685848] Lustre: server umount lustre-OST0000 complete [12879.569284] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [12898.127538] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [12940.026283] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 18:09:54 (1789078194) [12947.722914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12948.152421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12948.192261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12948.380392] Lustre: lustre-MDT0000: new disk, initializing [12948.514798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12948.557367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12953.162895] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12965.173742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12965.317366] Lustre: 384300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12965.411241] LustreError: 384310:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12965.420477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12965.437309] LustreError: 384310:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12965.443300] Lustre: Skipped 1 previous similar message [12965.582650] Lustre: lustre-MDT0001: new disk, initializing [12965.690563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12965.732751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12965.747528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12970.840735] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [12982.372353] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12989.296696] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12997.957196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12998.420682] Lustre: lustre-OST0000: new disk, initializing [12998.431579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12998.448268] Lustre: 386267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12998.586803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13005.981475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13005.996264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13006.123492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13006.178599] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [13016.215990] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13025.763046] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13025.775375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13025.796504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13025.801203] Lustre: Skipped 4 previous similar messages [13027.383337] Lustre: server umount lustre-MDT0000 complete [13031.454422] LustreError: 384292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789078288 with bad export cookie 17876531005951563568 [13031.458444] LustreError: 385111:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13031.460898] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13031.471341] LustreError: 384292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13031.953202] Lustre: server umount lustre-MDT0001 complete [13046.193785] Lustre: server umount lustre-OST0000 complete [13076.968192] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [13087.630410] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [13118.695950] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12861 sec ========= 18:12:53 (1789078373) [13120.393291] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:12:55 (1789078375) === [13123.682549] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:12:58 (1789078378) === [13167.382964] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [13170.695548] Key type lgssc unregistered [13170.943587] LNet: 392631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13170.953808] LNetError: 392631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13170.969239] LNet: Removed LNI 192.168.206.127@tcp [13171.814620] Key type .llcrypt unregistered [13171.817103] Key type ._llcrypt unregistered