[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 460535100 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002363] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009146] LSM: Security Framework initializing [ 0.010058] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012077] *** VALIDATE selinux *** [ 0.021202] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026759] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028141] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029131] *** VALIDATE tmpfs *** [ 0.031352] *** VALIDATE proc *** [ 0.032294] *** VALIDATE cgroup *** [ 0.034006] *** VALIDATE cgroup2 *** [ 0.035299] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039026] Speculative Store Bypass: Vulnerable [ 0.041819] debug: unmapping init [mem 0xffffffff83e59000-0xffffffff83e60fff] [ 0.043223] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044749] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045036] ... version: 2 [ 0.046020] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050019] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052312] rcu: Hierarchical SRCU implementation. [ 0.054514] smp: Bringing up secondary CPUs ... [ 0.055626] x86: Booting SMP configuration: [ 0.056030] .... node #0, CPUs: #1 #2 #3 [ 0.059619] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131288] node 0 deferred pages initialised in 67ms [ 0.135130] devtmpfs: initialized [ 0.136368] x86/mm: Memory block size: 128MB [ 0.139991] gcov: version magic: 0x41383552 [ 0.142749] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147303] pinctrl core: initialized pinctrl subsystem [ 0.149158] [ 0.149738] ************************************************************* [ 0.152013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153008] ** ** [ 0.155012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157015] ** ** [ 0.159011] ** This means that this kernel is built to expose internal ** [ 0.161010] ** IOMMU data structures, which may compromise security on ** [ 0.163008] ** your system. ** [ 0.164008] ** ** [ 0.166008] ** If you see this message and you are not debugging the ** [ 0.167007] ** kernel, report this immediately to your vendor! ** [ 0.168009] ** ** [ 0.170013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171006] ************************************************************* [ 0.173616] NET: Registered protocol family 16 [ 0.174287] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180091] cpuidle: using governor menu [ 0.181527] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183299] PCI: Using configuration type 1 for base access [ 0.184090] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190094] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194022] cryptd: max_cpu_qlen set to 1000 [ 0.196096] ACPI: Added _OSI(Module Device) [ 0.196973] ACPI: Added _OSI(Processor Device) [ 0.197010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.197905] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201033] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206103] ACPI: Interpreter enabled [ 0.206914] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207008] ACPI: Using IOAPIC for interrupt routing [ 0.208079] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210293] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.219776] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.222040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.232483] acpiphp: Slot [2] registered [ 0.233063] acpiphp: Slot [5] registered [ 0.233809] acpiphp: Slot [6] registered [ 0.235064] acpiphp: Slot [7] registered [ 0.236101] acpiphp: Slot [8] registered [ 0.237103] acpiphp: Slot [9] registered [ 0.239106] acpiphp: Slot [10] registered [ 0.240000] acpiphp: Slot [3] registered [ 0.240000] acpiphp: Slot [4] registered [ 0.242105] acpiphp: Slot [11] registered [ 0.243124] acpiphp: Slot [12] registered [ 0.245104] acpiphp: Slot [13] registered [ 0.246130] acpiphp: Slot [14] registered [ 0.247111] acpiphp: Slot [15] registered [ 0.249132] acpiphp: Slot [16] registered [ 0.250057] acpiphp: Slot [17] registered [ 0.251081] acpiphp: Slot [18] registered [ 0.252118] acpiphp: Slot [19] registered [ 0.253072] acpiphp: Slot [20] registered [ 0.254006] acpiphp: Slot [21] registered [ 0.255000] acpiphp: Slot [22] registered [ 0.257139] acpiphp: Slot [23] registered [ 0.258070] acpiphp: Slot [24] registered [ 0.258943] acpiphp: Slot [25] registered [ 0.260133] acpiphp: Slot [26] registered [ 0.261056] acpiphp: Slot [27] registered [ 0.261882] acpiphp: Slot [28] registered [ 0.262050] acpiphp: Slot [29] registered [ 0.263083] acpiphp: Slot [30] registered [ 0.265084] acpiphp: Slot [31] registered [ 0.266057] PCI host bridge to bus 0000:00 [ 0.268019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.270025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.273038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.276062] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.278018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.280020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283768] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.286219] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.297011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.301057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.303017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.306017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.308015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.310580] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.313504] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.315032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.317948] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.324016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.339053] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.346925] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.354251] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.362017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.370018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.392025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.409165] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.418026] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.427028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.443029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.455675] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.461014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.465012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.478018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.489037] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.495017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.500018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.518020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.532671] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.544034] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.557027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.576028] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.589000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.595020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.604016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.620026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.631085] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.634632] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.637468] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.639342] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.641282] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.648239] iommu: Default domain type: Passthrough [ 0.650425] SCSI subsystem initialized [ 0.651166] ACPI: bus type USB registered [ 0.653112] usbcore: registered new interface driver usbfs [ 0.654064] usbcore: registered new interface driver hub [ 0.655056] usbcore: registered new device driver usb [ 0.657138] pps_core: LinuxPPS API ver. 1 registered [ 0.658008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.661073] PTP clock support registered [ 0.663140] EDAC MC: Ver: 3.0.0 [ 0.665203] PCI: Using ACPI for IRQ routing [ 0.668118] NetLabel: Initializing [ 0.670016] NetLabel: domain hash size = 128 [ 0.672016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.673111] NetLabel: unlabeled traffic allowed by default [ 0.676161] vgaarb: loaded [ 0.677312] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.679010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.684399] clocksource: Switched to clocksource kvm-clock [ 0.796529] VFS: Disk quotas dquot_6.6.0 [ 0.797768] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.799750] *** VALIDATE ramfs *** [ 0.800674] *** VALIDATE hugetlbfs *** [ 0.802661] pnp: PnP ACPI init [ 0.804800] pnp: PnP ACPI: found 6 devices [ 0.830588] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.833700] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.835481] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.837067] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.839019] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.840975] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.843199] NET: Registered protocol family 2 [ 0.845347] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.851355] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.855091] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.859752] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.863275] TCP: Hash tables configured (established 65536 bind 65536) [ 0.865518] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.868833] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.872103] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.875183] NET: Registered protocol family 1 [ 0.879626] RPC: Registered named UNIX socket transport module. [ 0.881760] RPC: Registered udp transport module. [ 0.883488] RPC: Registered tcp transport module. [ 0.885114] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.887629] NET: Registered protocol family 44 [ 0.889511] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.891803] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.893839] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.896315] PCI: CLS 0 bytes, default 64 [ 0.897996] Unpacking initramfs... [ 2.263418] debug: unmapping init [mem 0xffff95ec7cc54000-0xffff95ec7ffbffff] [ 2.267354] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.269544] software IO TLB: mapped [mem 0x00000000a2800000-0x00000000a6800000] (64MB) [ 2.272356] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.717183] Initialise system trusted keyrings [ 2.718341] Key type blacklist registered [ 2.719595] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.726450] zbud: loaded [ 2.728623] *** VALIDATE nfs *** [ 2.729402] *** VALIDATE nfs4 *** [ 2.730428] pstore: using deflate compression [ 2.733165] Platform Keyring initialized [ 2.818156] NET: Registered protocol family 38 [ 2.819613] Key type asymmetric registered [ 2.820501] Asymmetric key parser 'x509' registered [ 2.821542] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.823474] io scheduler mq-deadline registered [ 2.824523] io scheduler kyber registered [ 2.825423] io scheduler bfq registered [ 2.826430] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.828293] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.829899] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.831558] ACPI: Power Button [PWRF] [ 2.835191] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.839588] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.847211] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.851527] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.862952] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.890745] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.917774] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.921582] Non-volatile memory driver v1.3 [ 2.922932] Linux agpgart interface v0.103 [ 2.947273] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.949105] vda: detected capacity change from 0 to 76775424 [ 2.962036] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.965169] vdb: detected capacity change from 0 to 1073741824 [ 2.980699] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.983662] vdc: detected capacity change from 0 to 2621440000 [ 2.998471] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.001707] vdd: detected capacity change from 0 to 2621440000 [ 3.018453] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.021223] vde: detected capacity change from 0 to 4294967296 [ 3.033775] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.036267] vdf: detected capacity change from 0 to 4294967296 [ 3.041921] libphy: Fixed MDIO Bus: probed [ 3.045183] usbcore: registered new interface driver usbserial_generic [ 3.046410] usbserial: USB Serial support registered for generic [ 3.047520] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.050066] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.051048] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.052419] mousedev: PS/2 mouse device common for all mice [ 3.054295] rtc_cmos 00:05: RTC can wake from S4 [ 3.055910] rtc_cmos 00:05: registered as rtc0 [ 3.057040] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.059602] intel_pstate: CPU model not supported [ 3.059674] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.065510] hid: raw HID events driver (C) Jiri Kosina [ 3.066591] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.068176] usbcore: registered new interface driver usbhid [ 3.070907] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.071951] usbhid: USB HID core driver [ 3.072166] drop_monitor: Initializing network drop monitor service [ 3.076923] Initializing XFRM netlink socket [ 3.078576] NET: Registered protocol family 10 [ 3.081098] Segment Routing with IPv6 [ 3.081887] NET: Registered protocol family 17 [ 3.083957] mpls_gso: MPLS GSO support [ 3.089157] RAS: Correctable Errors collector initialized. [ 3.090753] AVX version of gcm_enc/dec engaged. [ 3.092460] AES CTR mode by8 optimization enabled [ 3.163235] sched_clock: Marking stable (3163211715, 0)->(4041212697, -878000982) [ 3.166337] registered taskstats version 1 [ 3.168265] Loading compiled-in X.509 certificates [ 3.170263] zswap: loaded using pool lzo/zbud [ 3.189537] Key type big_key registered [ 3.202038] Key type encrypted registered [ 3.203720] ima: No TPM chip found, activating TPM-bypass! [ 3.205868] ima: Allocated hash algorithm: sha1 [ 3.207904] ima: No architecture policies found [ 3.209691] evm: Initialising EVM extended attributes: [ 3.211600] evm: security.selinux [ 3.212709] evm: security.ima [ 3.213832] evm: security.capability [ 3.215136] evm: HMAC attrs: 0x1 [ 3.217213] rtc_cmos 00:05: setting system clock to 2026-09-16 23:38:24 UTC (1789601904) [ 3.222756] debug: unmapping init [mem 0xffffffff84e03000-0xffffffff84ffffff] [ 3.226559] debug: unmapping init [mem 0xffffffff83b82000-0xffffffff83e58fff] [ 3.237078] Write protecting the kernel read-only data: 28672k [ 3.240519] debug: unmapping init [mem 0xffffffff82203000-0xffffffff823fffff] [ 3.244738] debug: unmapping init [mem 0xffffffff82b14000-0xffffffff82bfffff] [ 3.276543] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.284425] systemd[1]: Detected virtualization kvm. [ 3.286086] systemd[1]: Detected architecture x86-64. [ 3.287811] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.314238] systemd[1]: No hostname configured. [ 3.316338] systemd[1]: Set hostname to . [ 3.318403] random: systemd: uninitialized urandom read (16 bytes read) [ 3.321264] systemd[1]: Initializing machine ID from random generator. [ 3.432401] random: systemd: uninitialized urandom read (16 bytes read) [ 3.434611] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.438774] random: systemd: uninitialized urandom read (16 bytes read) [ 3.440206] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.443231] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.944587] device-mapper: uevent: version 1.0.3 [ 3.947918] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.591035] virtio_net virtio0 ens2: renamed from eth0 [ 4.624847] random: fast init done [ 4.652742] scsi host0: ata_piix [ 4.684205] scsi host1: ata_piix [ 4.685561] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.686900] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.384630] dracut-initqueue[577]: RTNETLINK answers: File exists [ 9.719472] random: crng init done [ 9.720881] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.099048] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.251070] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.480611] SELinux: Disabled at runtime. [ 11.533895] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.543267] systemd[1]: Detected virtualization kvm. [ 11.546136] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.022957] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.026335] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.032162] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.035990] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.039942] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.049789] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.061098] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ 12.093227] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.491639] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.765552] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.782571] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.884217] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.898960] EDAC sbridge: Ver: 1.1.2 [ 14.626517] Key type dns_resolver registered [ 14.940340] NFS: Registering the id_resolver key type [ 14.942200] Key type id_resolver registered [ 14.943357] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg105-server login: [ 40.822254] libcfs: loading out-of-tree module taints kernel. [ 40.838080] Key type ._llcrypt registered [ 40.839131] Key type .llcrypt registered [ 40.890971] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 58.300941] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 60.068831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.087168] alg: No test for adler32 (adler32-zlib) [ 61.342205] hrtimer: interrupt took 3196049 ns [ 61.509649] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 62.371770] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 64.143294] Key type lgssc registered [ 66.077036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.220758] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 134.544885] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 149.287846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.302976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.547813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.578529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.641800] Lustre: lustre-MDT0000: new disk, initializing [ 150.730719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.751198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.278072] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.776510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.893868] Lustre: 6502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.925643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.929443] Lustre: Skipped 1 previous similar message [ 173.045466] Lustre: lustre-MDT0001: new disk, initializing [ 173.136215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.162867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.177642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.555329] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.243636] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.735578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.995344] Lustre: lustre-OST0000: new disk, initializing [ 191.998196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.002719] Lustre: 8437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.062631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.298563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.303695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.446512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.481306] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.789218] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.880731] Lustre: lustre-OST0001: new disk, initializing [ 213.882401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 213.885766] Lustre: 9510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.917063] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.479249] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.193581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 221.204143] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 221.298423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.408039] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.334851] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 246.050269] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.967251] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing yml_node [ 254.629761] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 257.392908] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 259.868718] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 261.701856] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 16 19:42:41 EDT 2026 [ 280.381783] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 282.112823] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.964194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.976739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.074859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.079372] Lustre: Skipped 3 previous similar messages [ 308.196501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.206766] Lustre: Skipped 6 previous similar messages [ 312.287143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.416124] Lustre: server umount lustre-MDT0000 complete [ 315.930955] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602217 with bad export cookie 1222560368199772190 [ 315.938894] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.942554] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 318.432295] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.437084] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.436692] LustreError: 6510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.437746] LustreError: 6510:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 318.445064] Lustre: Skipped 4 previous similar messages [ 318.480416] Lustre: Skipped 1 previous similar message [ 320.480349] LustreError: 12141:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.492643] LustreError: 12141:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 322.282392] Lustre: server umount lustre-MDT0001 complete [ 326.505365] Lustre: server umount lustre-OST0000 complete [ 330.939945] Lustre: server umount lustre-OST0001 complete [ 338.079574] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 348.464300] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 384.320137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.555254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.578171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 384.640195] Lustre: lustre-MDT0000: new disk, initializing [ 384.704658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 384.720150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.755064] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 398.962090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.087509] Lustre: 15063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.136363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.140549] Lustre: Skipped 1 previous similar message [ 399.194496] Lustre: lustre-MDT0001: new disk, initializing [ 399.277717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 399.304918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 399.317509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.904978] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 415.368735] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 422.020212] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 429.345645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.700611] Lustre: lustre-OST0000: new disk, initializing [ 429.703156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.708642] Lustre: 17033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.862280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.443390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.469287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.617176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 436.077138] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 446.690510] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 451.719811] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 451.911755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 456.446135] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 456.621294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 462.318098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.321265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.321551] LustreError: Skipped 1 previous similar message [ 462.332417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.357164] Lustre: Skipped 1 previous similar message [ 462.380446] Lustre: Skipped 3 previous similar messages [ 465.055372] Lustre: server umount lustre-OST0000 complete [ 471.011935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.017291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.041442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.595698] Lustre: server umount lustre-MDT0000 complete [ 477.664965] LustreError: 15076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.693573] LustreError: 15076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 480.239165] LustreError: 15056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602381 with bad export cookie 1222560368199773730 [ 480.248376] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 480.252444] LustreError: 15056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 480.583478] Lustre: server umount lustre-MDT0001 complete [ 485.074500] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 19:46:25 (1789602385) [ 491.958589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 492.296611] LustreError: 19300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.371781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.428296] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 497.642194] LustreError: 19300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.797700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.359939] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 520.015560] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 528.619459] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 536.802764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 536.998738] LustreError: 21121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.040837] LustreError: 21121:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 544.249576] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 553.800316] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 557.782451] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 566.752976] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.756747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 566.768914] Lustre: Skipped 2 previous similar messages [ 566.782907] Lustre: Skipped 4 previous similar messages [ 570.384680] Lustre: server umount lustre-MDT0000 complete [ 571.874988] LustreError: 19296:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.899647] LustreError: 19296:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 573.924101] LustreError: 19282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602475 with bad export cookie 1222560368199774612 [ 573.931178] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.236563] Lustre: server umount lustre-MDT0001 complete [ 587.024995] Lustre: server umount lustre-OST0000 complete [ 605.537623] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 616.450651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.763765] LustreError: 23685:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.862575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.866025] Lustre: Skipped 2 previous similar messages [ 621.038408] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 629.642630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.310511] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 638.184730] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 645.192061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 652.752390] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 661.877508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 662.075133] Lustre: lustre-OST0001: new disk, initializing [ 662.078796] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 662.087232] Lustre: 25997:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 663.659176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 663.673385] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 663.815733] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 671.293348] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 683.272528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 688.024780] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 705.002251] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 705.002251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 705.002302] Lustre: Skipped 1 previous similar message [ 705.008301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 705.025743] Lustre: Skipped 1 previous similar message [ 705.072717] Lustre: Skipped 3 previous similar messages [ 709.729554] Lustre: server umount lustre-MDT0000 complete [ 710.127230] LustreError: 23680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 710.140995] LustreError: 23680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 714.014332] LustreError: 23667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602615 with bad export cookie 1222560368199775459 [ 714.019080] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.025082] LustreError: 23667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 714.401051] Lustre: server umount lustre-MDT0001 complete [ 729.734374] Lustre: server umount lustre-OST0000 complete [ 731.360175] Lustre: 3635:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602616/real 1789602616] req@ffff95eceacf7b80 x1876533668205184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789602632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 734.320799] Lustre: server umount lustre-OST0001 complete [ 754.585404] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 766.250865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 766.846625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.851553] Lustre: Skipped 3 previous similar messages [ 772.138214] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 781.902407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 787.448300] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 790.862956] Lustre: 30172:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 797.944976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.802327] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 815.414040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 822.695794] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 830.731071] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 834.376343] Lustre: 32051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 841.706037] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.716106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 841.730681] Lustre: Skipped 5 previous similar messages [ 841.745139] Lustre: Skipped 3 previous similar messages [ 848.070645] Lustre: server umount lustre-MDT0000 complete [ 851.944800] LustreError: 29023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 851.974811] LustreError: 29023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 852.301752] LustreError: 29005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602753 with bad export cookie 1222560368199776754 [ 852.303774] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 852.310726] LustreError: 29005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 852.750768] Lustre: server umount lustre-MDT0001 complete [ 867.587378] Lustre: server umount lustre-OST0000 complete [ 881.768404] Lustre: server umount lustre-OST0001 complete [ 890.957904] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 19:53:11 (1789602791) [ 918.091835] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 932.330615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.501280] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 947.141947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.435140] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 955.640811] Lustre: 36003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 962.474435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 970.414360] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 979.800696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 986.568383] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 995.973966] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1000.736931] Lustre: 37882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1009.122334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1009.140115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.189054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.196734] Lustre: Skipped 4 previous similar messages [ 1014.701028] Lustre: server umount lustre-MDT0000 complete [ 1019.087801] LustreError: 37884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602920 with bad export cookie 1222560368199777860 [ 1019.089980] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.097428] LustreError: 37884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1019.627223] Lustre: server umount lustre-MDT0001 complete [ 1034.098945] Lustre: server umount lustre-OST0000 complete [ 1036.960132] Lustre: 3633:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602922/real 1789602922] req@ffff95ebc52e7100 x1876533668308096/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789602938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1038.472702] Lustre: server umount lustre-OST0001 complete [ 1047.736842] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 19:55:47 (1789602947) [ 1070.716223] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1083.660088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.400089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.409890] Lustre: Skipped 7 previous similar messages [ 1090.950515] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1100.955159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.685245] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1108.122439] Lustre: 41191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1113.313835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1120.670166] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1125.934985] LustreError: 41545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.960046] LustreError: 41545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 1130.871952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1138.110887] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1146.523786] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1150.560116] Lustre: 43069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1151.695656] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1151.706768] Lustre: Skipped 1 previous similar message [ 1157.091929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.134733] Lustre: Skipped 6 previous similar messages [ 1161.994723] Lustre: server umount lustre-MDT0000 complete [ 1165.911237] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603067 with bad export cookie 1222560368199778966 [ 1165.915350] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1165.922030] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1172.390405] Lustre: server umount lustre-MDT0001 complete [ 1180.844570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.489499] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1200.509220] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1204.550165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.267226] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1210.379905] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1210.397393] Lustre: Skipped 1 previous similar message [ 1217.009970] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1219.135569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1225.602331] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1227.440113] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1235.946151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1240.909922] Lustre: server umount lustre-MDT0000 complete [ 1245.838199] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603147 with bad export cookie 1222560368199780100 [ 1245.842661] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.849304] LustreError: 40024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1246.319663] Lustre: server umount lustre-MDT0001 complete [ 1253.742453] Lustre: server umount lustre-OST0000 complete [ 1257.725573] Lustre: server umount lustre-OST0001 complete [ 1266.727729] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 19:59:26 (1789603166) [ 1280.817705] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1291.622807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1298.123956] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1308.398231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.197519] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1319.087478] Lustre: 48830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1326.945305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1334.551225] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1344.421241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1352.414651] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1361.496820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1366.522618] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1380.840294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1380.852231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.864913] Lustre: Skipped 23 previous similar messages [ 1386.893399] Lustre: server umount lustre-MDT0000 complete [ 1390.997239] LustreError: 50727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603292 with bad export cookie 1222560368199780891 [ 1391.006350] LustreError: 50727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1391.014498] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1391.330698] Lustre: server umount lustre-MDT0001 complete [ 1397.982228] Lustre: server umount lustre-OST0000 complete [ 1402.440599] Lustre: server umount lustre-OST0001 complete [ 1414.284328] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 20:01:54 (1789603314) [ 1417.158369] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1419.329947] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 20:01:59 (1789603319) [ 1431.925653] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 1442.185900] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1466.652092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.003981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1467.209872] Lustre: lustre-MDT0000: new disk, initializing [ 1467.397631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1474.207675] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1486.384114] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1496.912995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.324474] Lustre: lustre-OST0000: new disk, initializing [ 1497.331831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1497.341306] Lustre: Skipped 1 previous similar message [ 1497.349744] Lustre: 54924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1498.459701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1498.474954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1498.525260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1505.068043] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1515.403699] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1518.381887] Lustre: DEBUG MARKER: create test files [ 1560.033183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.048204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1560.050844] Lustre: Skipped 15 previous similar messages [ 1562.357638] Lustre: server umount lustre-OST0000 complete [ 1567.129146] Lustre: server umount lustre-MDT0000 complete [ 1580.835659] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1584.264965] Key type lgssc unregistered [ 1584.584789] LNet: 56552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.592129] LNetError: 56552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.604693] LNet: Removed LNI 192.168.201.105@tcp [ 1585.727153] Key type .llcrypt unregistered [ 1585.729535] Key type ._llcrypt unregistered [ 1616.585555] Key type ._llcrypt registered [ 1616.598200] Key type .llcrypt registered [ 1616.702637] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1618.114723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1618.198788] alg: No test for adler32 (adler32-zlib) [ 1619.353520] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1619.656070] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1621.330564] Key type lgssc registered [ 1622.660591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1630.426247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1630.472540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.061135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.536532] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1643.075516] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1650.375749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1650.987848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.158945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:101 to 0x240000400:129) [ 1657.309509] Lustre: 59274:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1657.321693] Lustre: 59274:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1657.330456] Lustre: 59274:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1657.342402] Lustre: 59274:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1657.353394] Lustre: 59274:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1657.367897] Lustre: 59274:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1658.095957] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1666.353270] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1671.748758] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1677.632085] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1685.382965] Lustre: server umount lustre-OST0000 complete [ 1699.618911] Lustre: server umount lustre-MDT0000 complete [ 1714.228839] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1716.921221] Key type lgssc unregistered [ 1717.377739] LNet: 60882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.383591] LNetError: 60882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.413479] LNet: Removed LNI 192.168.201.105@tcp [ 1718.848571] Key type .llcrypt unregistered [ 1718.854956] Key type ._llcrypt unregistered [ 1752.453694] Key type ._llcrypt registered [ 1752.455972] Key type .llcrypt registered [ 1752.581187] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1753.890990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1753.965512] alg: No test for adler32 (adler32-zlib) [ 1755.208809] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1755.515993] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1757.185498] Key type lgssc registered [ 1758.356472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1764.719487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1764.743329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.117869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.019569] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1784.017777] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1792.683319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1792.986365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.076070] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1810.196355] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1811.372654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1811.477060] Lustre: 63719:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1811.486979] Lustre: 63719:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1811.491963] Lustre: 63719:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1811.499644] Lustre: 63719:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1811.510249] Lustre: 63719:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1811.522469] Lustre: 63719:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1822.709133] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1862.627034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.630865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1862.644050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.753462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1868.771081] Lustre: server umount lustre-OST0000 complete [ 1873.133392] Lustre: server umount lustre-MDT0000 complete [ 1885.408744] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 1888.796047] Key type lgssc unregistered [ 1889.123683] LNet: 65321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.141415] LNetError: 65321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.165191] LNet: Removed LNI 192.168.201.105@tcp [ 1890.220694] Key type .llcrypt unregistered [ 1890.225173] Key type ._llcrypt unregistered [ 1911.028651] Key type ._llcrypt registered [ 1911.032679] Key type .llcrypt registered [ 1911.143092] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 1929.369984] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 1931.213791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1931.314399] alg: No test for adler32 (adler32-zlib) [ 1932.466211] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1932.803131] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 1934.543734] Key type lgssc registered [ 1936.037827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1979.703652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1979.728595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.028839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1981.060803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1981.146499] Lustre: lustre-MDT0000: new disk, initializing [ 1981.273537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.318637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1986.896927] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 1999.469333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.653846] Lustre: 68471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1999.783396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1999.785922] Lustre: Skipped 1 previous similar message [ 1999.837389] Lustre: lustre-MDT0001: new disk, initializing [ 1999.942450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1999.985048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1999.998996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2005.926561] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2020.622692] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2029.501259] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2036.695675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2036.951276] Lustre: lustre-OST0000: new disk, initializing [ 2036.954453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2036.958393] Lustre: 70443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2037.054932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2044.368760] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2046.032685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2046.052132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2046.159049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2055.869869] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2062.003832] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2062.227321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2068.187652] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2068.541653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2072.032264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2072.037707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2072.051052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2072.061258] Lustre: Skipped 1 previous similar message [ 2072.069724] Lustre: Skipped 1 previous similar message [ 2077.170433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2077.187330] Lustre: Skipped 1 previous similar message [ 2082.278592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2085.345652] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2085.474734] Lustre: server umount lustre-OST0000 complete [ 2092.008659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2092.019238] LustreError: Skipped 1 previous similar message [ 2092.025903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.055623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.065634] Lustre: Skipped 1 previous similar message [ 2097.632420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2097.636379] Lustre: Skipped 3 previous similar messages [ 2104.287497] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2104.477047] Lustre: server umount lustre-MDT0000 complete [ 2108.918311] LustreError: 68464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604010 with bad export cookie 6535296510067857812 [ 2108.932934] LustreError: 68464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2108.936925] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.280096] Lustre: server umount lustre-MDT0001 complete [ 2119.911693] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 20:13:39 (1789604019) [ 2131.860022] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2177.313736] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 2192.845130] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2241.446709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.829565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2241.851791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2241.972826] Lustre: lustre-MDT0000: new disk, initializing [ 2242.036192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.054538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2247.327738] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2260.524348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.714509] Lustre: 76492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2260.793822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2260.799352] Lustre: Skipped 1 previous similar message [ 2260.877533] Lustre: lustre-MDT0001: new disk, initializing [ 2260.966444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2261.031125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2261.040519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2266.049716] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2277.395324] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2284.626122] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2292.835621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2293.233877] Lustre: lustre-OST0000: new disk, initializing [ 2293.236640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2293.240586] Lustre: 78462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2293.383725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.428801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2298.457994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2298.591963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2300.404824] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2311.887968] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2318.052381] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2318.358341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2324.568946] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2324.922174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2329.576695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2329.592074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2329.604074] Lustre: Skipped 1 previous similar message [ 2329.607501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2329.607511] Lustre: Skipped 2 previous similar messages [ 2333.657332] Lustre: server umount lustre-OST0000 complete [ 2339.814987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.834678] Lustre: Skipped 2 previous similar messages [ 2344.884911] Lustre: server umount lustre-MDT0000 complete [ 2344.937168] LustreError: 76502:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.974370] LustreError: 76502:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2349.027871] LustreError: 77710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.046904] LustreError: 77710:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2349.204367] LustreError: 77294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604250 with bad export cookie 6535296510067858694 [ 2349.207585] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2349.214660] LustreError: 77294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2349.744091] Lustre: server umount lustre-MDT0001 complete [ 2361.371977] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 20:17:40 (1789604260) [ 2369.228901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.658896] LustreError: 80874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.733315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.153605] LustreError: 80875:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2375.278205] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2379.243951] LustreError: 80874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.436504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2391.480571] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2400.297407] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2406.572731] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2413.975540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2414.471331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.477035] Lustre: Skipped 1 previous similar message [ 2422.125635] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2432.415178] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2436.914681] Lustre: 83548:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting cc4d48b0-9d3a-4ae8-9749-fd688ef32023 at adminstrative request [ 2441.193198] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2441.224404] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2451.428873] Lustre: MGS: Client cc4d48b0-9d3a-4ae8-9749-fd688ef32023 (at 0@lo) reconnecting [ 2451.468205] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x5ab2098ce871e078 to 0x5ab2098ce871e318 [ 2451.490993] Lustre: MGC192.168.201.105@tcp: Connection restored to 0@lo (at 0@lo) [ 2475.362800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2475.563626] Lustre: lustre-OST0001: new disk, initializing [ 2475.567560] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2475.572046] Lustre: 83944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2481.241272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2481.252133] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2481.349767] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2482.526835] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2493.882314] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2502.125875] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2502.133558] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2502.140598] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2502.140607] Lustre: Skipped 3 previous similar messages [ 2502.140985] Lustre: Skipped 1 previous similar message [ 2502.155370] LustreError: Skipped 1 previous similar message [ 2503.340919] Lustre: server umount lustre-OST0001 complete [ 2507.241962] LustreError: 82695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.253204] LustreError: 82695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2513.485864] Lustre: server umount lustre-OST0000 complete [ 2517.472249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.490781] Lustre: Skipped 3 previous similar messages [ 2523.716249] Lustre: server umount lustre-MDT0000 complete [ 2527.714079] LustreError: 80874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.740085] LustreError: 80874:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2528.850995] LustreError: 80855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604430 with bad export cookie 6535296510067860248 [ 2528.874307] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.228837] Lustre: server umount lustre-MDT0001 complete [ 2540.332486] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 20:20:39 (1789604439) [ 2542.237752] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2544.587148] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 20:20:44 (1789604444) [ 2546.548278] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2548.803555] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 20:20:48 (1789604448) [ 2550.952497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2552.642920] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 20:20:52 (1789604452) [ 2554.418064] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2556.750572] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 20:20:56 (1789604456) [ 2558.806383] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2569.119379] loop: module loaded [ 2569.141743] loop0: detected capacity change from 0 to 4294967296 [ 2569.200829] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2571.954435] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2578.031643] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2582.943488] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2612.509746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2612.557845] LustreError: 88072:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2612.561418] LustreError: 88072:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2612.580916] LustreError: 88072:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2612.601430] LustreError: 88072:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2633.320165] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 2645.978877] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 2685.272543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.541231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2685.561765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2685.619493] Lustre: lustre-MDT0000: new disk, initializing [ 2685.677770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.681521] Lustre: Skipped 1 previous similar message [ 2685.690989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2690.111674] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2702.539368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.734295] Lustre: 90938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2702.964420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2708.595903] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2719.127640] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2727.453391] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2736.274976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.777484] Lustre: 92911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2744.862702] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2745.947685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2746.169272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2757.066932] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2762.290221] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2762.506686] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2767.921207] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2768.242791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2771.937819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2771.942027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.952189] LustreError: Skipped 1 previous similar message [ 2771.966172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.966182] Lustre: Skipped 9 previous similar messages [ 2771.988333] Lustre: Skipped 1 previous similar message [ 2776.754991] Lustre: server umount lustre-OST0000 complete [ 2787.509449] Lustre: server umount lustre-MDT0000 complete [ 2791.122286] LustreError: 90931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604692 with bad export cookie 6535296510067860864 [ 2791.133346] LustreError: 90931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2791.141695] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2791.579819] Lustre: server umount lustre-MDT0001 complete [ 2800.074612] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 20:25:00 (1789604700) [ 2807.590996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.145614] LustreError: 95324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.319439] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2822.956595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.396612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2823.399517] Lustre: Skipped 3 previous similar messages [ 2827.648702] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2835.447812] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2841.927500] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2849.690994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2857.036846] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2865.508765] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2869.749502] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2869.979383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2875.727822] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2875.922898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2883.555063] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2883.843214] Lustre: lustre-OST0001: new disk, initializing [ 2883.845307] Lustre: Skipped 2 previous similar messages [ 2883.849536] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2883.852038] Lustre: Skipped 3 previous similar messages [ 2883.865420] Lustre: 98693:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2892.620622] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 2892.956308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2892.974570] Lustre: Skipped 2 previous similar messages [ 2892.989344] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2893.171678] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2904.303653] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2910.905586] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2911.362226] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2917.504304] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2917.764711] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2939.324879] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2965.667655] Lustre: Failing over lustre-MDT0000 [ 2966.034643] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 2966.037683] Lustre: Skipped 5 previous similar messages [ 2966.124679] Lustre: server umount lustre-MDT0000 complete [ 2967.015503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2967.026697] LustreError: Skipped 1 previous similar message [ 2967.038524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.056365] Lustre: Skipped 2 previous similar messages [ 2967.071833] LustreError: 96063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.095492] LustreError: 96063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2986.464930] Lustre: 66694:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789604871/real 1789604871] req@00000000728e493d x1876535630130816/t0(0) o400->MGC192.168.201.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789604887 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2986.487896] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2993.965971] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2993.968075] LDISKFS-fs (dm-0): recovery complete [ 2993.976357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2996.127190] LustreError: 101512:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2996.137224] LustreError: 101512:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000007caa998e x1876535630136576/t0(0) o250->MGC192.168.201.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1789604897 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2996.155912] LustreError: 101512:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2997.124382] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3001.855711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3001.871767] Lustre: 95320:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3002.381095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3002.474832] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3002.496737] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 2 previous similar messages [ 3002.591273] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3002.693772] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3002.911882] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3003.246188] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3003.262227] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 3003.587818] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3003.601107] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 3003.629126] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3003.645572] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 6 previous similar messages [ 3004.288031] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3004.300252] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 3004.623505] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3004.634424] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 3005.935670] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3005.954276] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 6 previous similar messages [ 3006.593680] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3006.602301] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 3006.927181] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3006.937584] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 3010.163085] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3010.174460] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 3010.818324] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3010.832406] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 3011.159203] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3011.165420] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 3018.178210] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3018.189933] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 3018.843830] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3018.862412] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 3019.183191] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3019.189212] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 3034.464778] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3034.477079] Lustre: 101545:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 50 previous similar messages [ 3035.121277] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3035.133245] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 3035.457145] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3035.467402] LustreError: 101545:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 50 previous similar messages [ 3061.167161] Lustre: 101545:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3061.180917] Lustre: 101545:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3061.200402] Lustre: 101545:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 66816296-56f5-4a48-a90f-c6db1b9c8410@192.168.201.5@tcp [ 3061.219942] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3061.230346] Lustre: 101545:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@00000000ab0f8218 x1876535609990272/t0(8589934780) o35->66816296-56f5-4a48-a90f-c6db1b9c8410@192.168.201.5@tcp:538/0 lens 392/0 e 0 to 0 dl 1789604973 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3061.267700] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3061.274166] Lustre: lustre-MDT0000: Denying connection for new client 66816296-56f5-4a48-a90f-c6db1b9c8410 (at 192.168.201.5@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3061.307541] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3061.335097] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3061.345820] Lustre: Skipped 2 previous similar messages [ 3061.355859] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3061.403434] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:48 to 0x2c0000401:545) [ 3061.405642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 3077.601568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3077.607330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3077.614935] Lustre: Skipped 3 previous similar messages [ 3080.614487] Lustre: server umount lustre-OST0000 complete [ 3091.921961] Lustre: server umount lustre-OST0001 complete [ 3097.569436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.577616] Lustre: Skipped 9 previous similar messages [ 3111.391375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3111.628489] Lustre: server umount lustre-MDT0000 complete [ 3113.953572] LustreError: 96063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.970271] LustreError: 96063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 45 previous similar messages [ 3115.817284] LustreError: 97167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605017 with bad export cookie 6535296510067912076 [ 3115.817547] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3115.832683] LustreError: 97167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3116.220248] Lustre: server umount lustre-MDT0001 complete [ 3125.468369] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 20:30:25 (1789605025) [ 3134.286784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.292288] Lustre: *** cfs_fail_loc=197, val=0*** [ 3149.856273] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3154.975477] LustreError: 103522:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 3160.117297] Lustre: *** cfs_fail_loc=197, val=0*** [ 3160.124938] Lustre: Skipped 65 previous similar messages [ 3180.703435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.709221] Lustre: Skipped 3 previous similar messages [ 3187.783541] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3196.658402] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3199.193195] Lustre: server umount lustre-OST0000 complete [ 3208.015780] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 20:31:47 (1789605107) [ 3230.164055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.630574] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3245.126179] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3249.547506] LustreError: 105173:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3249.563832] LustreError: 105173:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3249.583825] LustreError: 105173:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3250.106824] Lustre: server umount lustre-MDT0000 complete [ 3288.607561] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3299.676302] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3345.571873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.769796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3345.789724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3345.868961] Lustre: lustre-MDT0000: new disk, initializing [ 3345.983880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3352.327292] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3365.171916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3365.428418] Lustre: 109354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3366.077096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3371.519562] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3381.414171] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3387.821221] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3395.505525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.729608] Lustre: 111321:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3402.442762] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3403.935445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3414.515636] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3421.026774] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3421.470248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3429.997156] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3430.362611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3434.465493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3434.472536] LustreError: Skipped 4 previous similar messages [ 3434.477506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3434.495138] Lustre: Skipped 5 previous similar messages [ 3434.506474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3434.516078] Lustre: Skipped 6 previous similar messages [ 3447.263343] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3447.623540] Lustre: server umount lustre-OST0000 complete [ 3460.578602] LustreError: 112029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.612711] LustreError: 112029:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3460.635300] Lustre: server umount lustre-MDT0000 complete [ 3464.728430] LustreError: 109344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605366 with bad export cookie 6535296510067922772 [ 3464.731083] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3464.738972] LustreError: 109344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3465.295609] Lustre: server umount lustre-MDT0001 complete [ 3475.535065] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 20:36:15 (1789605375) [ 3488.355772] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 3491.804526] Key type lgssc unregistered [ 3492.062158] LNet: 113811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3492.069788] LNetError: 113811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3492.091415] LNet: Removed LNI 192.168.201.105@tcp [ 3492.963852] Key type .llcrypt unregistered [ 3492.966376] Key type ._llcrypt unregistered [ 3513.015895] Key type ._llcrypt registered [ 3513.021958] Key type .llcrypt registered [ 3513.147788] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3529.258625] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3530.230322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3530.244703] alg: No test for adler32 (adler32-zlib) [ 3531.541441] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 3531.756486] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 3533.407160] Key type lgssc registered [ 3534.643985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3584.986103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3585.014076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.462309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3586.622451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3586.786713] Lustre: lustre-MDT0000: new disk, initializing [ 3586.893041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3586.937261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3593.741500] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3604.533604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3604.745188] Lustre: lustre-OST0000: new disk, initializing [ 3604.747138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3604.749360] Lustre: Skipped 1 previous similar message [ 3604.751754] Lustre: 117988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3604.806604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.328848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3611.347412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3611.392878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3612.011683] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3624.303386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3624.471744] Lustre: lustre-OST0001: new disk, initializing [ 3624.477676] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3624.483904] Lustre: 118851:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3624.617105] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3632.426875] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3634.375714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3634.388937] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3634.423954] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3644.914281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3644.914918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3644.932570] Lustre: Skipped 1 previous similar message [ 3644.952030] Lustre: Skipped 1 previous similar message [ 3649.722099] Lustre: server umount lustre-MDT0000 complete [ 3657.418688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3662.145333] LustreError: 117170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605563 with bad export cookie 4127051593507546699 [ 3662.149546] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3662.151814] LustreError: 117170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3662.273433] Lustre: server umount lustre-OST0000 complete [ 3666.170572] Lustre: server umount lustre-OST0001 complete [ 3674.856266] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 20:39:34 (1789605574) [ 3699.138633] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 20:39:59 (1789605599) [ 3712.541948] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3735.847964] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 20:40:35 (1789605635) [ 3780.669341] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 3790.452347] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 3828.238688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3828.515664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3828.537463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3828.634475] Lustre: lustre-MDT0000: new disk, initializing [ 3828.750078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3828.772725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3833.844372] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3844.715561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3844.822419] Lustre: 126604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3844.866628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3844.869893] Lustre: Skipped 1 previous similar message [ 3844.943121] Lustre: lustre-MDT0001: new disk, initializing [ 3845.031510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3845.072619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3845.091640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3849.939901] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3860.433924] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3867.094350] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3875.701495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3876.117916] Lustre: lustre-OST0000: new disk, initializing [ 3876.131895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3876.139155] Lustre: 128573:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3876.345398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.594865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3883.599634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3883.729992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3884.110269] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3894.747846] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3900.009800] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3900.268920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3905.904542] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3906.219617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3909.611794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3909.617292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3909.633296] LustreError: Skipped 1 previous similar message [ 3909.643510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3909.653444] Lustre: Skipped 1 previous similar message [ 3909.676818] Lustre: Skipped 1 previous similar message [ 3914.920484] Lustre: server umount lustre-OST0000 complete [ 3919.842346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.850704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3919.862256] Lustre: Skipped 1 previous similar message [ 3919.879776] Lustre: Skipped 1 previous similar message [ 3924.961136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3924.964544] Lustre: Skipped 1 previous similar message [ 3925.925874] Lustre: server umount lustre-MDT0000 complete [ 3929.931608] LustreError: 128571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605831 with bad export cookie 4127051593507547924 [ 3929.949839] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3929.952503] LustreError: 128571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3930.372737] Lustre: server umount lustre-MDT0001 complete [ 3940.269704] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 20:44:00 (1789605840) [ 3947.957784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3948.422861] LustreError: 130989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3948.449162] LustreError: 130989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3948.526131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3953.281608] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3953.645353] LustreError: 130990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3957.749393] LustreError: 130989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.418883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3967.559653] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 3975.743301] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3982.029753] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3990.878865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3991.366386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.370778] Lustre: Skipped 1 previous similar message [ 3999.844198] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4009.402196] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4013.990832] Lustre: *** cfs_fail_loc=159, val=0*** [ 4014.004401] Lustre: Skipped 2 previous similar messages [ 4030.522057] Lustre: lustre-MDT0000: Client d67ee237-5014-4094-b5c6-e00da6eb1622 (at 192.168.201.5@tcp) reconnecting [ 4032.481822] Lustre: *** cfs_fail_loc=159, val=0*** [ 4032.484915] Lustre: Skipped 3 previous similar messages [ 4047.943435] Lustre: lustre-MDT0000: Client d67ee237-5014-4094-b5c6-e00da6eb1622 (at 192.168.201.5@tcp) reconnecting [ 4069.347667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4069.352033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.356346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4069.380449] Lustre: Skipped 1 previous similar message [ 4072.036557] Lustre: server umount lustre-OST0000 complete [ 4079.589967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.592664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.610520] Lustre: Skipped 1 previous similar message [ 4079.630433] Lustre: Skipped 2 previous similar messages [ 4082.944402] Lustre: server umount lustre-MDT0000 complete [ 4084.711445] LustreError: 131719:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.740847] LustreError: 131719:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4087.091894] LustreError: 131727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605988 with bad export cookie 4127051593507548806 [ 4087.099592] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4087.105921] LustreError: 131727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4087.636660] Lustre: server umount lustre-MDT0001 complete [ 4099.014031] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4101.065446] Key type lgssc unregistered [ 4101.286687] LNet: 134765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4101.292050] LNetError: 134765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4101.317301] LNet: Removed LNI 192.168.201.105@tcp [ 4101.964617] Key type .llcrypt unregistered [ 4101.966327] Key type ._llcrypt unregistered [ 4136.708958] Key type ._llcrypt registered [ 4136.710855] Key type .llcrypt registered [ 4136.830452] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4138.295777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4138.475292] alg: No test for adler32 (adler32-zlib) [ 4140.192932] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 4140.733899] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4142.543200] Key type lgssc registered [ 4143.572666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4152.657731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4152.673816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.329965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.891862] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4164.576396] LustreError: 136177:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4164.605191] LustreError: 136177:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4169.703959] LustreError: 136176:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4170.218357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.781378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4176.776181] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4186.589361] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4193.427925] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4202.298961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4202.873971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.016462] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4220.228352] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4224.385662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4235.901386] Lustre: *** cfs_fail_loc=159, val=0*** [ 4252.205475] Lustre: lustre-MDT0000: Client 71aa0dc0-cec7-4e3a-a152-d36d526de4d7 (at 192.168.201.5@tcp) reconnecting [ 4254.959637] Lustre: *** cfs_fail_loc=159, val=0*** [ 4270.653631] Lustre: lustre-MDT0001: Client 71aa0dc0-cec7-4e3a-a152-d36d526de4d7 (at 192.168.201.5@tcp) reconnecting [ 4272.300921] Lustre: *** cfs_fail_loc=159, val=0*** [ 4272.312845] Lustre: Skipped 3 previous similar messages [ 4288.589921] Lustre: lustre-MDT0001: Client 71aa0dc0-cec7-4e3a-a152-d36d526de4d7 (at 192.168.201.5@tcp) reconnecting [ 4304.917349] Lustre: 139249:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4308.219772] Lustre: *** cfs_fail_loc=159, val=0*** [ 4308.223868] Lustre: Skipped 3 previous similar messages [ 4323.902603] Lustre: lustre-MDT0000: Client fdcf5a20-1c01-4044-840b-bd2e01164322 (at 192.168.201.5@tcp) reconnecting [ 4325.950270] Lustre: *** cfs_fail_loc=159, val=0*** [ 4325.958539] Lustre: Skipped 15 previous similar messages [ 4342.328139] Lustre: lustre-MDT0000: Client fdcf5a20-1c01-4044-840b-bd2e01164322 (at 192.168.201.5@tcp) reconnecting [ 4344.039538] Lustre: 139539:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4362.737793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4362.755567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4362.762216] Lustre: Skipped 1 previous similar message [ 4367.844135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4367.852142] Lustre: Skipped 1 previous similar message [ 4368.855371] Lustre: server umount lustre-OST0000 complete [ 4375.522520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4375.532687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4375.543242] Lustre: Skipped 1 previous similar message [ 4375.545553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4378.082525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.083732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4379.621381] LustreError: 139875:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000057afa483) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 4379.807692] Lustre: server umount lustre-MDT0000 complete [ 4383.205284] LustreError: 136176:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.223654] LustreError: 136176:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4384.302859] LustreError: 139370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606285 with bad export cookie 17506343211899193101 [ 4384.317363] LustreError: 139370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4384.320772] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4384.700344] Lustre: server umount lustre-MDT0001 complete [ 4396.669790] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4399.378286] Key type lgssc unregistered [ 4399.675291] LNet: 140627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4399.683377] LNetError: 140627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4399.701284] LNet: Removed LNI 192.168.201.105@tcp [ 4400.642673] Key type .llcrypt unregistered [ 4400.645227] Key type ._llcrypt unregistered [ 4428.192259] Key type ._llcrypt registered [ 4428.195649] Key type .llcrypt registered [ 4428.362503] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4429.504130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4429.693877] alg: No test for adler32 (adler32-zlib) [ 4430.772715] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 4431.034920] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4432.703497] Key type lgssc registered [ 4433.893795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4442.898444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4442.927936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.333838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.826955] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4453.868374] LustreError: 142037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4453.896645] LustreError: 142037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4458.978457] LustreError: 142038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4459.695941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.401443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4466.011333] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4475.135470] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4482.320609] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4490.350927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4490.744389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.588907] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4505.088752] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4507.122215] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4515.352390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4523.861597] Lustre: 144647:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4527.693984] Lustre: 144695:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4529.486546] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4530.774336] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4535.539875] Lustre: 144852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4536.435948] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4537.434439] Lustre: 144950:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4546.022514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4546.038774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4546.042137] LustreError: Skipped 1 previous similar message [ 4546.046866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4546.092739] Lustre: Skipped 1 previous similar message [ 4550.256619] Lustre: server umount lustre-OST0000 complete [ 4556.256305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4556.272384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4556.281289] Lustre: Skipped 2 previous similar messages [ 4560.714444] Lustre: server umount lustre-MDT0000 complete [ 4561.385686] LustreError: 142033:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.399171] LustreError: 142033:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4565.098391] LustreError: 142019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606466 with bad export cookie 13665633029809217392 [ 4565.104461] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4565.113031] LustreError: 142019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4565.630725] Lustre: server umount lustre-MDT0001 complete [ 4576.678856] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4579.697515] Key type lgssc unregistered [ 4579.918922] LNet: 146006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4579.928223] LNetError: 146006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4579.950159] LNet: Removed LNI 192.168.201.105@tcp [ 4580.812927] Key type .llcrypt unregistered [ 4580.815904] Key type ._llcrypt unregistered [ 4607.816729] Key type ._llcrypt registered [ 4607.818922] Key type .llcrypt registered [ 4607.929639] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4609.224176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4609.263926] alg: No test for adler32 (adler32-zlib) [ 4610.486683] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 4610.791091] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4612.543853] Key type lgssc registered [ 4613.939728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4623.576502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4623.617547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4625.323451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.894676] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4635.627143] LustreError: 147421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4635.644054] LustreError: 147421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4640.745032] LustreError: 147420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4641.673524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.264115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4647.703561] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4656.389373] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4662.957521] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4670.760136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4671.327557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.569549] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4686.475861] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4687.743749] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4694.921546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4703.198302] Lustre: *** cfs_fail_loc=159, val=0*** [ 4703.206495] Lustre: Skipped 1 previous similar message [ 4718.701320] Lustre: lustre-MDT0000: Client 57fc8d0a-b71b-4a39-ac5c-cde1d200c4f0 (at 192.168.201.5@tcp) reconnecting [ 4725.731341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.736386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4725.751922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4725.788269] Lustre: Skipped 1 previous similar message [ 4730.851844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4730.859251] Lustre: Skipped 1 previous similar message [ 4731.208041] Lustre: server umount lustre-OST0000 complete [ 4735.976016] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.993129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.001074] Lustre: Skipped 2 previous similar messages [ 4736.020628] Lustre: Skipped 1 previous similar message [ 4741.105413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4741.114368] Lustre: Skipped 1 previous similar message [ 4741.502437] Lustre: server umount lustre-MDT0000 complete [ 4745.740784] LustreError: 147402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606647 with bad export cookie 2784578328232885007 [ 4745.751117] LustreError: 147402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4745.751625] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4746.198734] Lustre: server umount lustre-MDT0001 complete [ 4759.144588] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4762.267534] Key type lgssc unregistered [ 4762.650794] LNet: 151107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4762.663273] LNetError: 151107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4762.689688] LNet: Removed LNI 192.168.201.105@tcp [ 4764.076230] Key type .llcrypt unregistered [ 4764.077796] Key type ._llcrypt unregistered [ 4791.085543] Key type ._llcrypt registered [ 4791.089586] Key type .llcrypt registered [ 4791.195282] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 4792.158455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4792.234206] alg: No test for adler32 (adler32-zlib) [ 4793.321943] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 4793.632272] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 4795.336434] Key type lgssc registered [ 4796.656754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4803.730968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4803.757808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.091498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.273810] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4815.347485] LustreError: 152516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4815.364598] LustreError: 152516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4820.452743] LustreError: 152517:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4820.746985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4821.241882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4826.166292] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4834.697837] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4841.957490] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4849.750460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4850.088805] LustreError: 154344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4850.281600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4855.814533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4857.304324] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 4866.042234] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4866.245493] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4885.944450] Lustre: lustre-MDT0001: evicting 2b59420f-81c8-4d88-b606-f000fcc501bb (at 192.168.201.5@tcp) by administrative request [ 4917.216272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4917.216695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.226030] LustreError: Skipped 1 previous similar message [ 4917.228560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4917.267793] Lustre: Skipped 1 previous similar message [ 4921.678916] Lustre: server umount lustre-OST0000 complete [ 4927.459617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.473242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.480192] Lustre: Skipped 2 previous similar messages [ 4932.578269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4932.590390] Lustre: Skipped 1 previous similar message [ 4937.697501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4937.706430] Lustre: Skipped 1 previous similar message [ 4941.279197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4941.511276] Lustre: server umount lustre-MDT0000 complete [ 4942.817912] LustreError: 152511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.830433] LustreError: 152511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4945.462658] LustreError: 152499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606846 with bad export cookie 10836963975584784161 [ 4945.464791] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4945.479423] LustreError: 152499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4946.164876] Lustre: server umount lustre-MDT0001 complete [ 4958.776267] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 4962.307498] Key type lgssc unregistered [ 4962.648822] LNet: 156598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4962.660120] LNetError: 156598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4962.680605] LNet: Removed LNI 192.168.201.105@tcp [ 4963.886158] Key type .llcrypt unregistered [ 4963.888477] Key type ._llcrypt unregistered [ 5036.914658] Key type ._llcrypt registered [ 5036.917288] Key type .llcrypt registered [ 5037.018113] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5038.144815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5038.166282] alg: No test for adler32 (adler32-zlib) [ 5039.243885] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 5039.442740] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5041.159245] Key type lgssc registered [ 5041.941928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5050.228674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5050.264506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.244943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5057.068727] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5062.634648] LustreError: 159086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5062.656827] LustreError: 159086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5067.765709] LustreError: 159085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5068.159503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.570074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5073.853361] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5085.984354] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5093.448901] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5102.459643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5103.171949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5110.427692] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5113.437901] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5122.131171] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5122.609895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5132.774420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5132.783636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.795576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.808416] Lustre: Skipped 1 previous similar message [ 5132.817409] Lustre: Skipped 1 previous similar message [ 5137.897334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5137.908160] Lustre: Skipped 1 previous similar message [ 5138.628896] Lustre: server umount lustre-OST0000 complete [ 5145.569736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5145.580091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.590354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5148.130102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.131523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5148.156141] Lustre: Skipped 1 previous similar message [ 5149.493089] Lustre: server umount lustre-MDT0000 complete [ 5153.134226] LustreError: 159065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607054 with bad export cookie 10987815632323787374 [ 5153.144936] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5153.154747] LustreError: 159065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5153.432955] Lustre: server umount lustre-MDT0001 complete [ 5163.570976] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5166.309421] Key type lgssc unregistered [ 5166.611867] LNet: 162630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5166.621323] LNetError: 162630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5166.643341] LNet: Removed LNI 192.168.201.105@tcp [ 5167.453621] Key type .llcrypt unregistered [ 5167.455596] Key type ._llcrypt unregistered [ 5192.437509] Key type ._llcrypt registered [ 5192.443462] Key type .llcrypt registered [ 5192.562332] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5193.784281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5194.051412] alg: No test for adler32 (adler32-zlib) [ 5195.329850] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 5195.654751] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5197.327453] Key type lgssc registered [ 5198.376987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5215.106900] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 21:05:15 (1789607115) [ 5234.024592] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 5244.075593] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5294.041378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5294.100647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.411180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5295.462341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5295.618789] Lustre: lustre-MDT0000: new disk, initializing [ 5295.787701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5295.821343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5301.943698] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5313.586806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5313.718533] Lustre: 166894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5313.820615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5313.823132] Lustre: Skipped 1 previous similar message [ 5313.930841] Lustre: lustre-MDT0001: new disk, initializing [ 5314.093227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5314.152592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5314.159389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5319.190595] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5329.590028] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5337.623320] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5345.546647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5346.026426] Lustre: lustre-OST0000: new disk, initializing [ 5346.029668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5346.042481] Lustre: 168861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5346.159040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5353.639598] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5355.139722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5355.148199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5355.259615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5364.655319] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5375.985414] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.986655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5375.989051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5375.998122] Lustre: Skipped 1 previous similar message [ 5376.027725] LustreError: Skipped 1 previous similar message [ 5381.098447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5381.104246] Lustre: Skipped 2 previous similar messages [ 5386.215722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5386.226591] Lustre: Skipped 1 previous similar message [ 5389.279135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5389.457370] Lustre: server umount lustre-OST0000 complete [ 5395.938289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5395.952774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5395.967318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5399.876441] Lustre: server umount lustre-MDT0000 complete [ 5404.375646] LustreError: 166885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607305 with bad export cookie 12800998229152326246 [ 5404.382838] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5404.390340] LustreError: 166885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5404.696790] Lustre: server umount lustre-MDT0001 complete [ 5416.729298] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 5419.944398] Key type lgssc unregistered [ 5420.242500] LNet: 170768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5420.247715] LNetError: 170768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5420.273728] LNet: Removed LNI 192.168.201.105@tcp [ 5421.378783] Key type .llcrypt unregistered [ 5421.381149] Key type ._llcrypt unregistered [ 5449.912772] Key type ._llcrypt registered [ 5449.914750] Key type .llcrypt registered [ 5450.051521] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 5451.952348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5451.986245] alg: No test for adler32 (adler32-zlib) [ 5453.088647] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 5453.335787] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 5455.015322] Key type lgssc registered [ 5456.228577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5464.245616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5464.262185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5465.787360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5470.475987] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5475.298483] LustreError: 172178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5475.322240] LustreError: 172178:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5480.398991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.418164] LustreError: 172177:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.959823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5486.756149] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5495.890731] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5502.571444] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5511.325297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5511.909922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.537838] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 5528.208454] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5535.482422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5545.969661] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5545.982825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.990824] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5545.998470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.012428] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5547.099664] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5547.120553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.130845] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5547.142447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.156575] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5548.171724] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5548.193466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5548.201146] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5548.205445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.212487] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5549.235033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.380932] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5550.385209] Lustre: Skipped 1 previous similar message [ 5550.401113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.409430] Lustre: Skipped 1 previous similar message [ 5550.413590] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5550.418519] Lustre: Skipped 1 previous similar message [ 5550.421057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.435269] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5550.440977] Lustre: Skipped 1 previous similar message [ 5551.366857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.126028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.335699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.321132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.416368] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5555.425678] Lustre: Skipped 4 previous similar messages [ 5555.439645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.448664] Lustre: Skipped 4 previous similar messages [ 5555.451951] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5555.454393] Lustre: Skipped 4 previous similar messages [ 5555.458237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.465567] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5555.468258] Lustre: Skipped 4 previous similar messages [ 5556.453780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.559692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.407798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.257629] LustreError: 172197:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5559.272514] LustreError: 172197:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5559.277475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.185946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.375917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.283356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.172280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.989258] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5563.992831] Lustre: Skipped 8 previous similar messages [ 5564.007637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5564.016112] Lustre: Skipped 8 previous similar messages [ 5564.024834] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5564.030203] Lustre: Skipped 8 previous similar messages [ 5564.039633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.053361] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5564.064650] Lustre: Skipped 8 previous similar messages [ 5565.073905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.106545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.146814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.041834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.111993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.273884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.267915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.153452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.043763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.953145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.697251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.911401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.297428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.197908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.326151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.256195] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5580.266271] Lustre: Skipped 15 previous similar messages [ 5580.306993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.324812] Lustre: Skipped 15 previous similar messages [ 5580.359611] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5580.364881] Lustre: Skipped 15 previous similar messages [ 5580.368288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.383976] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5580.387185] Lustre: Skipped 15 previous similar messages [ 5581.359356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.370953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.231803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.163308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.201713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.547880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.527876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.569966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.671299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.634986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.725906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.838085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.989809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.820831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.688677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.454316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.690174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.907312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.148423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.216925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.210629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.330843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.333108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.548831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.630193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.799343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.788788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.046578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.119781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.128465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.291505] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5613.298353] Lustre: Skipped 30 previous similar messages [ 5613.313073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.325907] Lustre: Skipped 30 previous similar messages [ 5613.336270] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5613.343312] Lustre: Skipped 30 previous similar messages [ 5613.347184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.358532] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5613.365738] Lustre: Skipped 30 previous similar messages [ 5614.572829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.698970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.783017] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.900287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.057088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.236055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.336830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.489882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.881630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.175681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.951261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.890721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.720263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.633716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.835690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.928191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.018516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.945787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.986425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.988661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.173228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.153276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.132463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.201541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.273798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.329262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.341735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.327430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.307628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.415316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.695650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.892796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.801709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.979200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.211956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.033851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.182334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.163963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.252637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.592743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.812402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.975922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.228471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.285045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.276575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.230069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.354897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.338216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.507804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.235832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.412392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.291196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.536312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.542993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.948783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.015702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.172608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.159083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.346425] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5677.356521] Lustre: Skipped 58 previous similar messages [ 5677.379339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5677.388415] Lustre: Skipped 58 previous similar messages [ 5677.392108] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5677.397458] Lustre: Skipped 58 previous similar messages [ 5677.404678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.415752] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5677.423679] Lustre: Skipped 58 previous similar messages [ 5678.556730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.867172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.243533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.345191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.947507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.786312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.235406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.712419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.341551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.067160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.412183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.923354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.948241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.122794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.652936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.542315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.881241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.833628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.873140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.844564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.740213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.918059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.487283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.157890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.097099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.486031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.682218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.104439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.995050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.977731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.702391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.681222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.430431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.869383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.954379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.452377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.857087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.554638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.627045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.945458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.926519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.089956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.863359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.820714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.835860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.842896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.971462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.922285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.098075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.099755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.158333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.310735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.374564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.573336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.760175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.075163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.984533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.103869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.105510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.381250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.373827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.606530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.673295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.764547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.937219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.998088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.079400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.009736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.901206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.838855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.018164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.041617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.036572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.059236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.133040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.359917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.602723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.538796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.767752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.920679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.860287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.018588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.135307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.293912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.416392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.616879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.663116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.909515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.834499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.862711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.871922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.902141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.868618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.789783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.768651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.747654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.729902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.693167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.551587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.532676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.422828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.430294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.117701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.946731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.605664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.739661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.998425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.897932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.162350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.133866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.119846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.109268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.410102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.397953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.472568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.324414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.396415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.150420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.979206] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5805.985182] Lustre: Skipped 118 previous similar messages [ 5805.993589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5805.999159] Lustre: Skipped 118 previous similar messages [ 5806.002313] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5806.005436] Lustre: Skipped 118 previous similar messages [ 5806.008046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.022959] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5806.032283] Lustre: Skipped 118 previous similar messages [ 5806.898539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.832272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.710895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.616811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.364399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.384773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.503598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.452718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.490490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.372991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.730992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.626419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.610196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.576752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.558979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.540324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.580959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.379340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.377552] LustreError: 172197:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5824.387447] LustreError: 172197:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5824.393965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.196565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.164046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.311915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.257456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.361513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.201422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.070606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.172963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.216221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.055879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.942598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.081000] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.175957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.350498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.621876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.453584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.495425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.511427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.562798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.498789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.461189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.385947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.317963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.229806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.215454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.207379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.371282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.430495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.617449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.667240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.791856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.837938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.190397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.482487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.615164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.796682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.042643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.128050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.449321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.563699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.509699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.113413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.263168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.614435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.047521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.260472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.557640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.910503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.261313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.680632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.047880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.486215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.705402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.881206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.197626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.002683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.883388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.055893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.575213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.476925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.237989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.461499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.682592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.875160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.689868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.663830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.917182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.346389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.287295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.739157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.498380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.728569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.877407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.218678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.228597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.339828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.209115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.331351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.237433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.322118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.320926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.435168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.338263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.386258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.627400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.867475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.056305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.230382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.303639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.602076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.493091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.546076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.658915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.133843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.400046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.585310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.762791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.936415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.172385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.320475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.484822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.412264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.572617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.570817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.483121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.575697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.488811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.630880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.615491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.653521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.537664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.534895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.415531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.395175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.435504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.325905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.286757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.361112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.265932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.199344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.272625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.271718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.320462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.191568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.333673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.538400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.477319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.880289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.983944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.113849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.174874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.093916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.198147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.107236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.413544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.377157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.486180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.449240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.560523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.525266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.491113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.431801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.603496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.563639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.617312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.678717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.471708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.545759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.520722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.804945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.800423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.700708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.299913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.908644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.152048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.277342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.353355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.414823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.385996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.337229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.143748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.121377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.191557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.973782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.089768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.106703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.071807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.984926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.144979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.451451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.638427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.712960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.955367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.589815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.817470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.779926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.123852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.542609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.658593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.741889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.699414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.979381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.208160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.268951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.369854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.795995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.992606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.185166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.139749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.108356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.056865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.175353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.022629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.939587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.976499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.879620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.791970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.976694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.883289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.903822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.874245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.895891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.061708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.387315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.588432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.492103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.665775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.826952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.859267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.078985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.321687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.419943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.619483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.765026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.943466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.039378] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6062.052049] Lustre: Skipped 234 previous similar messages [ 6062.090250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6062.110353] Lustre: Skipped 234 previous similar messages [ 6062.126447] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6062.136986] Lustre: Skipped 234 previous similar messages [ 6062.148264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.183221] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6062.200769] Lustre: Skipped 234 previous similar messages [ 6063.334642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.440840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.634040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.686445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.717062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.942667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.998263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.166378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.437715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.776105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.722849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.855517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.946939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.122607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.270914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.652333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.742619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.673921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.824181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.946876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.198619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.354170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.773423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.204840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.595724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.746420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.775719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.828494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.986926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.905881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.002708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.067414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.199155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.383214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.486454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.526893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.841211] LustreError: 172197:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6104.860363] LustreError: 172197:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6104.873648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.971776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.133649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.301073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.292638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.342381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.278980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.338859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.309842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.496843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.697807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.805768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.974927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.213539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.419748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.693424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.779275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.127408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.179056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.111874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.814478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.729133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.934925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.155679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.518686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.687652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.296955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.360873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.509946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.618149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.724210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.928581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.089465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.070063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.295677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.658600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.033847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.381868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.690090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.917335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.642102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.969698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.937645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.942379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.169846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.172403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.190167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.179166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.237160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.565258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.482582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.621175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.791943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.814352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.817230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.872898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.001391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.196038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.333534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.453748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.603347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.809801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.940301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.053585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.211387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.566104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.798350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.779852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.095402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.188259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.281027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.564375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.630902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.609684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.527832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.548818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.553364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.506745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.674093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.593966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.708378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.577986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.437355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.247395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.308525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.612046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.702695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.457816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.773571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.977154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.164680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.413683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.568308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.452645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.502501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.322950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.210575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.204882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.235155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.146574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.234854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.160538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.091560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.966638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.086726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.246295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.059533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.195529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.235041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.410900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.563524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.728483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.724446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.962056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.934951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.066500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.092982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.253252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.380514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.539319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.785592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.156665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.467842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.590072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.807716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.226474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.479698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.742689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.244162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.447945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.699549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.785413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.124373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.513936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.339050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.319303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.879330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.238507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.912966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.795248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.653249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.210701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.154795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.701595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.000827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.340486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.305144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.467130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.375664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.420107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.506530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.481279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.462056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.426893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.906448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.851262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.937389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.090121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.300847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.467434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.841564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.489734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.711789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.859385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.042881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.127079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.448164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.514599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.592413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.409983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.211687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.368379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.781627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.549141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.932362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.554604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.875470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.982833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.174632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.627437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.757937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.195304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.381807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.941266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.846873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.784740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.884448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.817809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.514811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.421685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.323730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.071044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.922471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.964854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.962791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.514845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.777555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.569740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.523121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.668981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.806323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.923940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.097347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.351659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.914429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.119620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.134189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.144924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.381342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.533796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.594652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.570298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.749729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.782373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.970957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.172273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.439403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.588645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.706457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.624992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.605933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.645343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.590047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.947288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.313800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.397694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.531838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.891417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.060516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.310236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.364769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.621527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.924318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.340213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.586499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.208401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.668216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.147951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.247939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.475221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.819376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.735257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.783553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.668210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.819605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.230657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.764613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.826956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.144283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.491164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.487795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.249583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.381308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.722910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.032276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.221648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.667403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.832214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.167923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.325746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.335149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.596573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.733099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.889910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.919304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.362859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.368304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.576688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.839920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.968625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.311540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.530923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.747304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.049328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.322534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.484765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.668144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.166380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.331753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.402659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.335700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.429928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.504068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.389924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.602696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.534845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.721622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.715780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.744225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.643547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.826284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.948199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.925170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.948480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.840266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.826465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.945727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.946980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.903211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.198853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.364778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.414102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.354603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.320087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.331049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.379721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.289477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.214045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.141060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.964248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.141767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.262300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.498595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.619838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.646344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.531070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.438454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.538732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.726355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.975048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.071530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.213727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.208264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.310979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.384352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.504121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.829579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.892101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.202316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.356781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.344502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.459096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.499702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.643659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.757543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.673151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.673732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.593990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.603215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.496946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.527237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.493466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.632435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.603156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.675563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.829780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.908501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.947991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.847683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.688990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.737084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.758386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.636428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.772060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.523876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.574691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.539976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.521277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.390696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.394332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.474424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.462547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.868394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.893285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.918506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.910112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.968155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.009501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.066912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.129336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.126880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.071561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.107955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.971055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.940144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.085101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.067720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.049778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.910205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.804604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.711728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.566279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.397769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.453446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.426097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.421388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.401805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.265424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.238180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.036939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6543.057156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.142182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.026187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.103713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.711645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.812220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.922941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.839867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.921309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.951511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.120735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.077180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.445387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.737142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.958587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.035380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.326069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.596519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.880705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.799798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6565.039566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.203731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.499770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.579674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.736293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6570.903540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6572.253579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6573.382529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.353381] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6574.369163] Lustre: Skipped 454 previous similar messages [ 6574.387429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.396750] Lustre: Skipped 454 previous similar messages [ 6574.404431] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6574.409127] Lustre: Skipped 454 previous similar messages [ 6574.412311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.426057] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6574.438932] Lustre: Skipped 454 previous similar messages [ 6575.326382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.177685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.950351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6577.819206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6578.679295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6579.523201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.416831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.252663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.120228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.899550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.693966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6584.581696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6585.633734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6586.548949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.591573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6588.361385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.217614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6590.208522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6591.194549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.079956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.986461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.996364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6595.376211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6596.424389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.221450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.293579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6599.499344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6600.730721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6601.946983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6602.843300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6604.005857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6604.866249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6606.064803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6607.125852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6608.489184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.552187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6610.497832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6611.366353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6612.601868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.570745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6614.558729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6615.315979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6616.511891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6617.239715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6617.998577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6618.864256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6619.891824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6621.004557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6621.888178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.014157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.932904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6625.532427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6627.173510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6628.829500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6630.265427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6631.858819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6633.221291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6634.750563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6635.923370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6636.887937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6638.123178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6639.124093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6640.285950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6641.136624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6642.253696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6643.160937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7081.915580] LustreError: 174002:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 7081.923045] LustreError: 172197:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7742.435440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.453903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.456054] Lustre: Skipped 67 previous similar messages [ 7742.489024] Lustre: Skipped 1 previous similar message [ 7747.571467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7747.580499] Lustre: Skipped 1 previous similar message [ 7747.874554] Lustre: server umount lustre-OST0000 complete [ 7752.680055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7752.690515] Lustre: Skipped 1 previous similar message [ 7757.794797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7757.813497] Lustre: Skipped 1 previous similar message [ 7759.059468] Lustre: server umount lustre-MDT0000 complete [ 7762.913350] LustreError: 172178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7762.946681] LustreError: 172178:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7763.508213] LustreError: 172159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789609664 with bad export cookie 460916283563972248 [ 7763.524198] LustreError: 172159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7763.537142] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7764.148300] Lustre: server umount lustre-MDT0001 complete [ 7775.032857] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 7778.308410] Key type lgssc unregistered [ 7778.743663] LNet: 227489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7778.757409] LNetError: 227489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7778.783771] LNet: Removed LNI 192.168.201.105@tcp [ 7779.663815] Key type .llcrypt unregistered [ 7779.666719] Key type ._llcrypt unregistered [ 7805.379842] Key type ._llcrypt registered [ 7805.381769] Key type .llcrypt registered [ 7805.482456] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 7806.896794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7806.921611] alg: No test for adler32 (adler32-zlib) [ 7808.122403] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 7808.403178] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 7810.183253] Key type lgssc registered [ 7811.300501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7819.688576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7819.723352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.358847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7826.494823] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7831.526875] LustreError: 228904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7831.546177] LustreError: 228904:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7834.706266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.072229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7839.913808] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7846.180993] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7851.041956] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7858.080415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7858.709286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7865.460829] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7865.915647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7874.194801] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7881.457278] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7883.860169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.876724] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7883.902642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7883.917850] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7883.918664] LustreError: 228924:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7886.310647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7886.327553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7886.337227] Lustre: Skipped 1 previous similar message [ 7887.395085] Lustre: lustre-OST0000: Not available for connect from 192.168.201.5@tcp (stopping) [ 7891.445165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7891.454385] Lustre: Skipped 1 previous similar message [ 7892.275487] Lustre: server umount lustre-OST0000 complete [ 7903.960395] LustreError: 228924:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7928.028436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.574192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.128385] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 7940.843735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7940.873156] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7940.878434] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7945.011852] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7955.191796] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 7962.653721] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a3ac74ff800.ost_server_uuid 50 [ 7964.639595] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a3ac74ff800.ost_server_uuid in FULL state after 0 sec [ 7970.274413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7970.278325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.287186] Lustre: Skipped 1 previous similar message [ 7970.294972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7971.810351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7974.688083] Lustre: server umount lustre-OST0000 complete [ 7978.472796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7978.489706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7978.504507] Lustre: Skipped 1 previous similar message [ 7978.514932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7978.521463] Lustre: Skipped 2 previous similar messages [ 7984.561201] Lustre: server umount lustre-MDT0000 complete [ 7987.169746] LustreError: 228900:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.199107] LustreError: 228900:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7987.985541] LustreError: 228886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789609889 with bad export cookie 9835092379344079583 [ 7987.988682] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7987.995061] LustreError: 228886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7988.418939] Lustre: server umount lustre-MDT0001 complete [ 7998.567318] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8001.140118] Key type lgssc unregistered [ 8001.422638] LNet: 234115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8001.430780] LNetError: 234115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8001.456287] LNet: Removed LNI 192.168.201.105@tcp [ 8002.234625] Key type .llcrypt unregistered [ 8002.239357] Key type ._llcrypt unregistered [ 8034.248367] Key type ._llcrypt registered [ 8034.250849] Key type .llcrypt registered [ 8034.415430] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8035.309533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8035.357940] alg: No test for adler32 (adler32-zlib) [ 8036.544704] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8036.894384] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 8038.608797] Key type lgssc registered [ 8039.912347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8049.321482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8049.341722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8050.817505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8056.753927] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8061.426284] LustreError: 235840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8061.463081] LustreError: 235840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8066.539487] LustreError: 235839:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8067.519899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.728807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8071.543606] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8073.998869] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8080.600436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8080.842590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8086.232442] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8094.256072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8094.400581] Lustre: lustre-OST0001: new disk, initializing [ 8094.404689] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8094.411262] Lustre: 238148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8094.497519] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8096.584812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8101.198979] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8103.474938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8103.484578] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8103.575244] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8114.019883] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8124.111993] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8139.748681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8139.751376] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.761230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8139.789565] Lustre: Skipped 3 previous similar messages [ 8144.867224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8144.878029] Lustre: Skipped 6 previous similar messages [ 8144.991357] Lustre: server umount lustre-MDT0000 complete [ 8148.142704] LustreError: 236575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610049 with bad export cookie 5807051862068981560 [ 8148.161297] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8148.472268] Lustre: server umount lustre-MDT0001 complete [ 8161.922703] Lustre: server umount lustre-OST0000 complete [ 8176.521663] Lustre: server umount lustre-OST0001 complete [ 8199.815071] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8210.826340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.440523] LustreError: 241254:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8211.570671] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8216.286374] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8217.059334] LustreError: 241255:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8221.159827] LustreError: 241254:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.153214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.785610] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8230.363623] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8233.492246] Lustre: 242395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8240.626369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8249.496644] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8250.362805] LustreError: 242752:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8256.498033] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 8258.875832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8259.170833] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8259.183830] Lustre: Skipped 1 previous similar message [ 8264.182838] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8264.952897] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8272.207840] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8275.698466] Lustre: 244267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8287.201819] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8287.209951] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8287.221839] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8289.772522] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8289.775824] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.784629] Lustre: Skipped 1 previous similar message [ 8289.811172] Lustre: Skipped 1 previous similar message [ 8293.475568] Lustre: server umount mylustre-MDT0000 complete [ 8294.883316] LustreError: 241254:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.908575] LustreError: 241254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 8297.112977] LustreError: 241231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610198 with bad export cookie 5807051862068984367 [ 8297.129450] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8297.133283] LustreError: 241231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8297.606524] Lustre: server umount mylustre-MDT0001 complete [ 8312.365209] Lustre: server umount mylustre-OST0000 complete [ 8316.260121] Lustre: 235103:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789610201/real 1789610201] req@ffff95ebd79b5880 x1876542030503936/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789610217 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8316.297363] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8316.324488] Lustre: Skipped 1 previous similar message [ 8316.717919] Lustre: server umount mylustre-OST0001 complete [ 8337.405518] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8348.178479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8348.843730] LustreError: 246571:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8348.951172] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8354.870814] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8365.140721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.231153] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8372.960343] Lustre: 247713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8380.872988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8388.297559] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8394.554707] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49792) [ 8399.314342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8405.010823] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8409.472391] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8418.955713] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8422.743891] Lustre: 249589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8432.096142] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8432.105966] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.118492] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8432.123910] Lustre: Skipped 2 previous similar messages [ 8433.079217] Lustre: server umount tfs-MDT0000 complete [ 8435.723299] LustreError: 246571:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8435.757455] LustreError: 246571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 8437.655044] LustreError: 248092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610338 with bad export cookie 5807051862068986257 [ 8437.660890] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8437.677229] LustreError: 248092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8438.251160] Lustre: server umount tfs-MDT0001 complete [ 8452.933277] Lustre: server umount tfs-OST0000 complete [ 8456.863116] Lustre: 235104:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789610342/real 1789610342] req@ffff95ebdbe87800 x1876542030557824/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789610358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8456.893768] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.917208] Lustre: Skipped 3 previous similar messages [ 8457.278054] Lustre: server umount tfs-OST0001 complete [ 8481.494826] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8492.787115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8493.679820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8493.693616] Lustre: Skipped 3 previous similar messages [ 8499.231208] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8509.494046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8515.180169] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8518.662511] Lustre: 252979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8525.674777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8533.168755] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8538.248027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49824) [ 8543.415047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8549.374646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8551.826559] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8560.629221] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8565.197125] Lustre: 254862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8573.178676] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 22:01:12 (1789610472) [ 8591.846334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8591.850566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8591.865422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8597.354925] Lustre: server umount lustre-MDT0000 complete [ 8600.548623] LustreError: 251834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.579250] LustreError: 251834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 8601.609225] LustreError: 251813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610502 with bad export cookie 5807051862068987923 [ 8601.614150] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8608.183784] Lustre: server umount lustre-MDT0001 complete [ 8627.167265] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8627.220118] Lustre: server umount lustre-OST0000 complete [ 8631.193196] LustreError: 235100:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff95ebc4944a80 x1876542030620928/t0(0) o38->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8637.590599] Lustre: server umount lustre-OST0001 complete [ 8648.148510] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8660.040269] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 8703.205019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.521460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8703.569117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8703.724575] Lustre: lustre-MDT0000: new disk, initializing [ 8703.788162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8703.793996] Lustre: Skipped 3 previous similar messages [ 8703.837394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8710.672848] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8722.359117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8722.574724] Lustre: 258053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8722.674740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8722.681976] Lustre: Skipped 1 previous similar message [ 8722.763222] Lustre: lustre-MDT0001: new disk, initializing [ 8722.890606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8722.906561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8728.177970] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8738.775247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8739.116358] Lustre: lustre-OST0000: new disk, initializing [ 8739.121669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8739.132932] Lustre: 259638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8741.098385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8741.104038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8741.293030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8747.677989] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8759.212610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8759.344442] Lustre: lustre-OST0001: new disk, initializing [ 8759.348592] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8759.357632] Lustre: 260509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8760.780532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8760.797314] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8760.963208] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8766.580754] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 8779.211223] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 22:04:39 (1789610679) [ 8792.622521] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 22:04:52 (1789610692) [ 8807.904275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8807.911456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.916298] LustreError: Skipped 1 previous similar message [ 8807.920325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8807.920333] Lustre: Skipped 8 previous similar messages [ 8807.943339] Lustre: Skipped 6 previous similar messages [ 8822.239814] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8822.368074] Lustre: server umount lustre-OST0000 complete [ 8830.433046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8842.207712] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8842.777625] Lustre: server umount lustre-MDT0000 complete [ 8847.143700] LustreError: 258045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610748 with bad export cookie 5807051862068989477 [ 8847.147587] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8847.149820] LustreError: 258045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8847.856672] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8847.863327] Lustre: Skipped 14 previous similar messages [ 8853.630550] Lustre: server umount lustre-MDT0001 complete [ 8865.828940] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 8947.063553] Lustre: server umount lustre-OST0001 complete [ 8954.756580] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 8973.702719] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9020.308803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9020.754218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9020.791889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9020.915993] Lustre: lustre-MDT0000: new disk, initializing [ 9021.065543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9021.082062] Lustre: Skipped 3 previous similar messages [ 9021.127523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9025.833829] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9037.113717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.307628] Lustre: 265374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9037.407493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9037.414076] Lustre: Skipped 1 previous similar message [ 9037.482760] Lustre: lustre-MDT0001: new disk, initializing [ 9037.638124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9037.658251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9043.138448] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9054.794736] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9060.012714] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9067.047940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9067.497097] Lustre: 267343:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9068.635043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9068.774332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9075.232658] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9085.018990] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9094.634615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.648398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9094.650755] Lustre: Skipped 5 previous similar messages [ 9094.657898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9094.659776] LustreError: Skipped 1 previous similar message [ 9094.680079] Lustre: Skipped 2 previous similar messages [ 9098.700944] Lustre: server umount lustre-OST0000 complete [ 9109.047593] Lustre: server umount lustre-MDT0000 complete [ 9109.988766] LustreError: 265381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9110.019394] LustreError: 265381:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 9112.909257] LustreError: 265367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611014 with bad export cookie 5807051862068990947 [ 9112.912640] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9112.922206] LustreError: 265367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9113.275719] Lustre: server umount lustre-MDT0001 complete [ 9124.068616] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9127.225371] Key type lgssc unregistered [ 9127.629279] LNet: 269105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9127.656621] LNetError: 269105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9127.687960] LNet: Removed LNI 192.168.201.105@tcp [ 9128.720177] Key type .llcrypt unregistered [ 9128.723397] Key type ._llcrypt unregistered [ 9157.510284] Key type ._llcrypt registered [ 9157.513629] Key type .llcrypt registered [ 9157.629459] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9158.832208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9158.867144] alg: No test for adler32 (adler32-zlib) [ 9159.956303] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 9160.199383] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 9161.871657] Key type lgssc registered [ 9162.916840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9171.286624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9171.320376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9172.884872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9177.078272] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9178.088875] LustreError: 270561:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9178.115200] LustreError: 270561:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9183.200925] LustreError: 270562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9185.558455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.927846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9189.784963] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9197.479818] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9203.736585] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9210.529417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9210.856855] LustreError: 272383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9211.072592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9218.027589] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9218.561540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9226.220802] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9227.319608] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9231.840444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9231.850836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9231.863801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9233.892754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9233.895303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9233.902623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9233.936563] Lustre: Skipped 1 previous similar message [ 9235.578539] Lustre: server umount lustre-OST0000 complete [ 9242.083544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9242.094086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.116799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9244.130402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9244.131773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9249.251548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9249.259434] Lustre: Skipped 2 previous similar messages [ 9254.367296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9254.788817] Lustre: server umount lustre-MDT0000 complete [ 9258.968054] LustreError: 271296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611160 with bad export cookie 3779234402683628637 [ 9258.978342] LustreError: 271296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9258.979112] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9259.340269] Lustre: server umount lustre-MDT0001 complete [ 9269.116699] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9271.875939] Key type lgssc unregistered [ 9272.174619] LNet: 274148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9272.183303] LNetError: 274148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9272.206533] LNet: Removed LNI 192.168.201.105@tcp [ 9272.998996] Key type .llcrypt unregistered [ 9273.001101] Key type ._llcrypt unregistered [ 9289.313199] Key type ._llcrypt registered [ 9289.315886] Key type .llcrypt registered [ 9289.427113] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9290.494427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9290.628673] alg: No test for adler32 (adler32-zlib) [ 9291.767707] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 9292.027292] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 9293.719220] Key type lgssc registered [ 9294.675243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9303.706678] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9303.723391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9319.200718] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9324.319340] LustreError: 275342:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.105@tcp: failed processing log, type 4: rc = -110 [ 9351.007456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9358.243646] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9367.505910] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9369.829401] Lustre: server umount lustre-OST0000 complete [ 9386.493834] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [ 9389.102649] Key type lgssc unregistered [ 9389.461609] LNet: 276973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9389.475187] LNetError: 276973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9389.511445] LNet: Removed LNI 192.168.201.105@tcp [ 9390.565726] Key type .llcrypt unregistered [ 9390.566741] Key type ._llcrypt unregistered [ 9427.834729] Key type ._llcrypt registered [ 9427.841941] Key type .llcrypt registered [ 9427.955858] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9429.219715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9429.246438] alg: No test for adler32 (adler32-zlib) [ 9430.425219] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 9430.693424] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [ 9432.359751] Key type lgssc registered [ 9433.613652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9446.890335] loop0: detected capacity change from 0 to 204800000 [ 9447.135423] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9447.143787] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9447.448712] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9448.489363] loop0: detected capacity change from 0 to 204800000 [ 9448.666594] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9448.675886] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9449.132846] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9450.199262] loop0: detected capacity change from 0 to 204800000 [ 9450.370192] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9450.377223] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9450.485350] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9451.449891] loop0: detected capacity change from 0 to 204800000 [ 9451.611670] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9452.903557] loop0: detected capacity change from 0 to 204800000 [ 9452.938663] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9457.210136] loop0: detected capacity change from 0 to 204800000 [ 9457.247488] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9461.511718] loop0: detected capacity change from 0 to 204800000 [ 9461.546705] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9465.174881] loop0: detected capacity change from 0 to 204800000 [ 9465.197275] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9468.785857] loop0: detected capacity change from 0 to 204800000 [ 9468.884296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9468.926031] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9468.990524] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9468.999322] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9469.009482] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9469.017430] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9469.024382] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9469.032193] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9469.038458] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9469.044356] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9469.048603] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9469.053446] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9469.058684] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9469.063271] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9469.068450] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9469.073271] Lustre: 280359:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9474.491110] Lustre: server umount MGS complete [ 9475.276388] loop0: detected capacity change from 0 to 204800000 [ 9475.378854] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9475.905043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9475.955035] Lustre: lustre-MDT0000: Aborting client recovery [ 9476.912401] loop1: detected capacity change from 0 to 204800000 [ 9477.086204] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.241334] Lustre: 280792:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9477.259834] Lustre: 280792:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9477.272672] Lustre: 280792:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9477.288614] Lustre: 280792:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9477.297241] Lustre: 280792:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9477.304419] Lustre: 280792:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9477.311535] Lustre: 280792:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9477.327191] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9477.445949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9477.489755] Lustre: lustre-MDT0001: Aborting client recovery [ 9478.467442] loop2: detected capacity change from 0 to 204800000 [ 9478.659295] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9478.775610] Lustre: 280871:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9478.789509] Lustre: 280871:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9478.806536] Lustre: 280871:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9478.819950] Lustre: 280871:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9478.833911] Lustre: 280871:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9478.853941] Lustre: 280871:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9478.865381] Lustre: 280871:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9479.042363] Lustre: Found index 0 for lustre-OST0000, updating log [ 9479.071646] Lustre: 280889:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9479.138330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9479.169037] Lustre: lustre-OST0000: Aborting recovery [ 9480.207357] loop3: detected capacity change from 0 to 204800000 [ 9480.344767] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9480.463901] Lustre: 280954:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9480.485831] Lustre: 280954:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9480.498633] Lustre: 280954:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9480.513945] Lustre: 280954:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9480.530978] Lustre: 280954:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9480.541086] Lustre: 280954:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9480.551675] Lustre: 280954:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9480.569292] Lustre: Found index 1 for lustre-OST0001, updating log [ 9480.595259] Lustre: 280959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9480.668908] Lustre: lustre-OST0001: Aborting recovery [ 9485.800987] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9485.830242] LustreError: 280736:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9485.852886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9485.876439] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9485.907850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9486.823026] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9486.838222] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9489.006812] LustreError: 280703:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 623us: rc = -2 [ 9489.526177] LustreError: 280701:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 744us: rc = -2 [ 9489.543954] LustreError: 280701:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 15 previous similar messages [ 9502.176406] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9502.180615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9502.187534] Lustre: Skipped 2 previous similar messages [ 9502.205318] Lustre: Skipped 3 previous similar messages [ 9507.297623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9507.302912] Lustre: Skipped 3 previous similar messages [ 9508.216418] Lustre: server umount lustre-MDT0000 complete [ 9509.224710] LustreError: 280687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611410 with bad export cookie 4696405877693449662 [ 9509.244444] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9512.424113] LustreError: 280702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9512.425996] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.434861] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9512.448866] LustreError: 280702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9512.489087] Lustre: Skipped 2 previous similar messages [ 9514.468829] LustreError: 280700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9514.478661] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9514.515119] Lustre: Skipped 2 previous similar messages [ 9515.829743] Lustre: server umount lustre-MDT0001 complete [ 9531.359436] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9531.488109] Lustre: server umount lustre-OST0000 complete [ 9538.598094] Lustre: server umount lustre-OST0001 complete [ 9546.490528] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 22:17:26 (1789611446) [ 9569.336259] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9611.242383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9611.556746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9611.578321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9611.634380] Lustre: lustre-MDT0000: new disk, initializing [ 9611.709853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9611.712817] Lustre: Skipped 1 previous similar message [ 9611.737076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9616.401140] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9623.684145] Lustre: server umount lustre-MDT0000 complete [ 9641.448399] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 22:19:00 (1789611540) [ 9666.189676] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [ 9682.687088] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [ 9721.708464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.053221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9722.111044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9722.116955] Lustre: Skipped 1 previous similar message [ 9722.235718] Lustre: lustre-MDT0000: new disk, initializing [ 9722.311037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9722.347992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9727.491491] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9736.862841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.949965] Lustre: 288084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9737.018636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9737.028383] Lustre: Skipped 1 previous similar message [ 9737.120193] Lustre: lustre-MDT0001: new disk, initializing [ 9737.236666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9737.288136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9737.291572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9741.533582] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9751.057366] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9756.839463] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9763.471586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9763.819926] Lustre: lustre-OST0000: new disk, initializing [ 9763.824937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9763.831425] Lustre: 290052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9763.934387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9770.088264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9770.100931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9770.197697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9770.268722] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9779.181766] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9783.692799] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9793.861316] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9804.467563] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9804.475643] Lustre: Skipped 1 previous similar message [ 9847.270048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9847.273655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9847.284774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9847.301723] Lustre: Skipped 1 previous similar message [ 9847.324196] Lustre: Skipped 1 previous similar message [ 9857.525530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9857.546120] Lustre: Skipped 3 previous similar messages [ 9861.087280] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9861.353386] Lustre: server umount lustre-OST0000 complete [ 9867.749036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9871.656290] Lustre: server umount lustre-MDT0000 complete [ 9872.869376] LustreError: 289181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.886657] LustreError: 289181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9875.590417] LustreError: 288077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611776 with bad export cookie 4696405877693452630 [ 9875.591253] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9875.600935] LustreError: 288077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9876.220826] Lustre: server umount lustre-MDT0001 complete [ 9885.999610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9891.377989] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9897.243273] Lustre: server umount MGS complete [ 9909.562972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9910.220698] LustreError: 293875:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9910.253875] LustreError: 293887:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9910.308114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9915.363122] LustreError: 293874:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9916.145161] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9924.891452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9930.101547] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9938.039589] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9944.564483] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9952.249477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9960.354727] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [ 9968.232862] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10001.387050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10001.390865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10001.394639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10001.394647] Lustre: Skipped 2 previous similar messages [10001.395685] LustreError: Skipped 1 previous similar message [10001.412666] Lustre: Skipped 2 previous similar messages [10004.361391] Lustre: server umount lustre-OST0000 complete [10011.624956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10011.642122] Lustre: Skipped 1 previous similar message [10022.367131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10022.600397] Lustre: server umount lustre-MDT0000 complete [10025.952539] LustreError: 293875:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.965902] LustreError: 293875:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10026.592215] LustreError: 293854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611927 with bad export cookie 4696405877693454569 [10026.595330] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10026.603962] LustreError: 293854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10026.965876] Lustre: server umount lustre-MDT0001 complete [10039.456409] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10042.375753] Key type lgssc unregistered [10042.681301] LNet: 297824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10042.691527] LNetError: 297824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10043.751832] LNet: Removed LNI 192.168.201.105@tcp [10044.740248] Key type .llcrypt unregistered [10044.743273] Key type ._llcrypt unregistered [10086.981151] Key type ._llcrypt registered [10086.985298] Key type .llcrypt registered [10087.069059] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [10102.229980] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10103.861950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10103.932073] alg: No test for adler32 (adler32-zlib) [10105.042584] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [10105.475496] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10107.231219] Key type lgssc registered [10108.482540] Lustre: Echo OBD driver; http://www.lustre.org/ [10149.072194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10149.107085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10150.364600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10150.427714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10150.511125] Lustre: lustre-MDT0000: new disk, initializing [10150.593591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10150.626427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10155.389167] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10168.471544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.588869] Lustre: 301779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10168.640401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10168.644560] Lustre: Skipped 1 previous similar message [10168.710988] Lustre: lustre-MDT0001: new disk, initializing [10168.793127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10168.816167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10168.825509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10173.278419] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10183.957001] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10191.297532] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10199.219922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10199.518834] Lustre: lustre-OST0000: new disk, initializing [10199.527288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10199.535978] Lustre: 303747:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10199.681518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10206.327690] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10208.316663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10208.325201] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10208.475577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10216.447315] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10223.472604] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10233.875958] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10239.992426] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10244.865370] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [10285.542193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10285.542949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10285.559444] LustreError: Skipped 1 previous similar message [10285.575115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10285.577204] Lustre: Skipped 1 previous similar message [10290.663064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10290.671036] Lustre: Skipped 1 previous similar message [10295.787042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10295.791329] Lustre: Skipped 1 previous similar message [10298.849760] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10299.199521] Lustre: server umount lustre-OST0000 complete [10306.017926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10306.021835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10306.052940] Lustre: Skipped 2 previous similar messages [10306.073359] Lustre: Skipped 1 previous similar message [10311.138565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10311.143121] Lustre: Skipped 1 previous similar message [10317.281511] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10317.630761] Lustre: server umount lustre-MDT0000 complete [10322.324170] LustreError: 305255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789612223 with bad export cookie 7651937464293785777 [10322.331233] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10322.336380] LustreError: 305255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10322.647907] Lustre: server umount lustre-MDT0001 complete [10332.340633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10335.668321] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10340.323985] Lustre: server umount MGS complete [10352.995330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.366124] LustreError: 307569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10353.380710] LustreError: 307569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10353.385387] LustreError: 307582:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10353.469823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10358.178294] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10358.755150] LustreError: 307570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10362.850256] LustreError: 307569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10366.430850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10366.666552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10370.999168] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10379.478772] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10385.344943] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10392.066105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10392.430904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10398.663257] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10406.191583] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10434.547462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10434.565029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10434.605787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10434.618315] Lustre: Skipped 2 previous similar messages [10436.580420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10436.597391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10440.325930] Lustre: server umount lustre-OST0000 complete [10446.820115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10446.833441] Lustre: Skipped 1 previous similar message [10451.074589] Lustre: server umount lustre-MDT0000 complete [10451.949910] LustreError: 307564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.968087] LustreError: 307564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10454.803202] LustreError: 307550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789612356 with bad export cookie 7651937464293787604 [10454.812191] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10454.823249] LustreError: 307550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10455.191072] Lustre: server umount lustre-MDT0001 complete [10467.442834] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [10471.257908] Key type lgssc unregistered [10471.601218] LNet: 311512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10471.615236] LNetError: 311512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10471.636648] LNet: Removed LNI 192.168.201.105@tcp [10472.765537] Key type .llcrypt unregistered [10472.768666] Key type ._llcrypt unregistered [10498.327655] Key type ._llcrypt registered [10498.330376] Key type .llcrypt registered [10498.439324] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [10515.409355] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10517.221491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10517.370581] alg: No test for adler32 (adler32-zlib) [10518.748472] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [10519.114541] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [10520.911293] Key type lgssc registered [10522.161208] Lustre: Echo OBD driver; http://www.lustre.org/ [10550.126972] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [10567.906193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10567.933921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10569.183320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10569.213708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10569.293724] Lustre: lustre-MDT0000: new disk, initializing [10569.376845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10569.393508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10574.323311] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10585.380386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10585.750717] Lustre: lustre-OST0000: new disk, initializing [10585.754245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10585.758976] Lustre: Skipped 1 previous similar message [10585.768221] Lustre: 315565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10585.828755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10592.182692] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [10594.384561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10594.395996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10594.422851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11513.826147] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.843808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11514.483244] Lustre: server umount lustre-MDT0000 complete [11518.043268] LustreError: 314750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789613419 with bad export cookie 16380257404012987489 [11518.047347] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11518.050496] LustreError: 314750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11518.191242] Lustre: server umount lustre-OST0000 complete [11551.427842] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [11565.661814] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11609.770040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.017983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11610.037892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11610.104929] Lustre: lustre-MDT0000: new disk, initializing [11610.268085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11610.288585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11615.353599] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11627.551093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11627.706523] Lustre: 319920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11627.752370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11627.754980] Lustre: Skipped 1 previous similar message [11627.836821] Lustre: lustre-MDT0001: new disk, initializing [11627.928572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11627.950576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11627.959854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11633.591972] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11645.792075] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11652.384698] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11660.317663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11660.675601] Lustre: lustre-OST0000: new disk, initializing [11660.679936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11660.689594] Lustre: 321892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11660.814777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11667.795746] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11669.097744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11669.104810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11669.223663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11677.374478] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11682.852499] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11683.089323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11688.170515] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11688.436149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11695.076478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11695.078358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11695.086838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11695.106907] Lustre: Skipped 1 previous similar message [11696.912577] Lustre: server umount lustre-OST0000 complete [11704.801146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11704.809788] LustreError: Skipped 1 previous similar message [11704.820183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11704.862989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11704.865440] Lustre: Skipped 1 previous similar message [11708.418776] Lustre: server umount lustre-MDT0000 complete [11713.462392] LustreError: 319914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789613614 with bad export cookie 16380257404015928308 [11713.472229] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11713.483849] LustreError: 319914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11713.951186] Lustre: server umount lustre-MDT0001 complete [11717.830156] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11719.991033] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 22:53:39 (1789613619) [11727.491821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.004389] LustreError: 324204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11728.034573] LustreError: 324204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11728.118040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11733.473596] LustreError: 324205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.586947] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11738.604530] LustreError: 324204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11743.531665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11743.712404] LustreError: 324205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11744.093788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11749.221520] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11758.641296] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11765.787939] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11773.491270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11773.734241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11780.435621] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11788.916815] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11796.762459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11796.887973] Lustre: lustre-OST0001: new disk, initializing [11796.901174] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11796.930610] Lustre: 327041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11797.107161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11805.380565] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11805.964312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11805.990428] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11806.093845] LustreError: 327419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11807.155377] LustreError: 327419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11807.170189] LustreError: 327419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11808.246157] LustreError: 327423:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11808.272213] LustreError: 327423:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11810.337542] LustreError: 327423:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11810.356896] LustreError: 327423:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11814.561843] LustreError: 327419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11814.568063] LustreError: 327419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [11820.595771] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11820.923044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11822.756848] LustreError: 327419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11822.767518] LustreError: 327419:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11827.927465] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a3ac4ef6800.ost_server_uuid 50 [11829.929797] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a3ac4ef6800.ost_server_uuid in FULL state after 0 sec [11835.836867] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a3ac4ef6800.ost_server_uuid 50 [11837.572180] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a3ac4ef6800.ost_server_uuid in IDLE state after 0 sec [11839.228418] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11854.391194] Lustre: lustre-OST0001: Not available for connect from 192.168.201.5@tcp (stopping) [11854.400422] Lustre: Skipped 2 previous similar messages [11854.834468] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11854.836947] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11854.853165] LustreError: Skipped 1 previous similar message [11854.899258] Lustre: Skipped 2 previous similar messages [11859.462184] Lustre: lustre-OST0001: Not available for connect from 192.168.201.5@tcp (stopping) [11859.471427] Lustre: Skipped 4 previous similar messages [11867.622884] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11867.633572] Lustre: Skipped 4 previous similar messages [11868.130622] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11868.220234] Lustre: server umount lustre-OST0001 complete [11869.713531] LustreError: 326025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.297618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11875.309956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11877.861091] LustreError: 326025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.881461] LustreError: 326025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11881.472235] Lustre: server umount lustre-OST0000 complete [11887.599315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11887.613274] LustreError: Skipped 1 previous similar message [11887.626348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11887.650304] Lustre: Skipped 1 previous similar message [11887.666518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11887.678277] Lustre: Skipped 4 previous similar messages [11900.383361] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11900.727787] Lustre: server umount lustre-MDT0000 complete [11903.459715] LustreError: 324205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11903.479560] LustreError: 324205:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11905.340659] LustreError: 324184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789613806 with bad export cookie 16380257404015929190 [11905.349110] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11905.363827] LustreError: 324184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11906.041604] Lustre: server umount lustre-MDT0001 complete [11917.830104] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [11921.360342] Key type lgssc unregistered [11921.794454] LNet: 329730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11921.802920] LNetError: 329730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11921.821757] LNet: Removed LNI 192.168.201.105@tcp [11922.691657] Key type .llcrypt unregistered [11922.692978] Key type ._llcrypt unregistered [11949.603833] Key type ._llcrypt registered [11949.610960] Key type .llcrypt registered [11949.808897] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [11951.345753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11951.448594] alg: No test for adler32 (adler32-zlib) [11952.709692] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [11953.063877] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [11954.847242] Key type lgssc registered [11956.315294] Lustre: Echo OBD driver; http://www.lustre.org/ [11965.015514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11965.034942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11966.430770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11972.016621] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [11976.683966] LustreError: 331164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11976.707100] LustreError: 331164:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11981.443668] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11981.792238] LustreError: 331163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.915042] LustreError: 331164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11989.723671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11990.310344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11996.127161] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12007.165158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12007.627824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12010.739353] LustreError: 332844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12010.771532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [12014.836725] LustreError: 332847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12014.872237] LustreError: 332847:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12014.917914] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12024.195538] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12025.316105] LustreError: 332844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12025.348474] LustreError: 332844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12032.612051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12032.884742] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12038.160592] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [12040.538709] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12049.293533] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12058.152804] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [12058.474800] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12169.203965] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12169.244273] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12169.251782] Lustre: Skipped 1 previous similar message [12173.278585] Lustre: server umount lustre-OST0001 complete [12174.325471] LustreError: 333014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12174.369549] LustreError: 333014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12184.037909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12184.044055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12184.052544] Lustre: Skipped 1 previous similar message [12184.058568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12186.452393] Lustre: server umount lustre-OST0000 complete [12194.795217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12194.800278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12194.819345] Lustre: Skipped 2 previous similar messages [12194.845070] Lustre: Skipped 2 previous similar messages [12197.920103] Lustre: server umount lustre-MDT0000 complete [12203.445519] LustreError: 334849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614104 with bad export cookie 5580200195266844402 [12203.451627] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12203.452063] LustreError: 334849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12204.229110] Lustre: server umount lustre-MDT0001 complete [12218.906111] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [12222.110888] Key type lgssc unregistered [12222.630528] LNet: 336122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12222.638762] LNetError: 336122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12222.676666] LNet: Removed LNI 192.168.201.105@tcp [12224.142286] Key type .llcrypt unregistered [12224.148141] Key type ._llcrypt unregistered [12279.705489] Key type ._llcrypt registered [12279.707744] Key type .llcrypt registered [12279.994778] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12282.294961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12282.399386] alg: No test for adler32 (adler32-zlib) [12283.576459] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [12283.931407] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [12285.647216] Key type lgssc registered [12287.293668] Lustre: Echo OBD driver; http://www.lustre.org/ [12299.323692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12299.341740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.740312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12304.788301] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12310.004474] LustreError: 338409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12310.039781] LustreError: 338409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12314.497150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12314.891768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12320.501597] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12324.017150] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12331.612824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12332.078794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12339.817796] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12340.282762] LustreError: 339896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12345.328322] LustreError: 339896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12346.361826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12349.855045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12350.140475] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12355.582478] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12357.432669] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12367.370431] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12376.223878] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12391.399839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12391.419517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12391.423711] Lustre: Skipped 2 previous similar messages [12391.441269] Lustre: Skipped 3 previous similar messages [12396.541666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12396.568199] Lustre: Skipped 3 previous similar messages [12396.991796] Lustre: server umount lustre-MDT0000 complete [12401.494692] LustreError: 338390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614302 with bad export cookie 2370084857025054991 [12401.504416] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12401.505115] LustreError: 338390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12401.632841] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12401.636207] LustreError: 340136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12401.644923] Lustre: Skipped 1 previous similar message [12401.650055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12401.656746] LustreError: 340136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12401.678642] Lustre: Skipped 1 previous similar message [12401.962364] Lustre: server umount lustre-MDT0001 complete [12408.820169] Lustre: server umount lustre-OST0000 complete [12413.002257] Lustre: server umount lustre-OST0001 complete [12431.989934] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12445.908302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12446.340303] LustreError: 343671:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12446.487042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12451.905818] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12455.913784] LustreError: 343672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12455.930520] LustreError: 343672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12463.752017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12464.298510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12469.509198] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12472.721268] Lustre: 344813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12480.667669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12481.107687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12488.734382] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12489.328874] LustreError: 345165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12489.361725] LustreError: 345165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12491.383314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12498.834738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12504.621379] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12506.775037] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12515.883586] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12520.585853] Lustre: 346684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12530.149121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12530.186779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12530.189255] Lustre: Skipped 2 previous similar messages [12530.215636] Lustre: Skipped 3 previous similar messages [12531.927350] Lustre: server umount lustre-MDT0000 complete [12535.267617] LustreError: 344676:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12535.285259] LustreError: 344676:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [12536.493502] LustreError: 343652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614437 with bad export cookie 2370084857025056692 [12536.500519] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12536.507036] LustreError: 343652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12536.834993] Lustre: server umount lustre-MDT0001 complete [12550.982120] Lustre: server umount lustre-OST0000 complete [12565.926831] Lustre: server umount lustre-OST0001 complete [12570.340894] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12572.904394] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12575.221952] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 23:07:54 (1789614474) [12602.662783] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12612.486247] loop0: detected capacity change from 0 to 1099511627776 [12613.462313] SGI XFS with ACLs, security attributes, quota, no debug enabled [12613.468488] *** VALIDATE xfs *** [12613.516994] XFS (loop0): Mounting V5 Filesystem [12613.585214] XFS (loop0): Ending clean mount [12615.425328] loop1: detected capacity change from 0 to 18691697672192 [12751.769862] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [12764.194266] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [12813.682788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12813.983426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12814.036467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12814.127264] Lustre: lustre-MDT0000: new disk, initializing [12814.218268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12814.226438] Lustre: Skipped 1 previous similar message [12814.258828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12819.429152] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12832.665303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12832.887545] Lustre: 351832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12832.976904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12832.982937] Lustre: Skipped 1 previous similar message [12833.081802] Lustre: lustre-MDT0001: new disk, initializing [12833.267808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12833.282691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12838.651597] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12850.470559] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12855.840841] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12862.958986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12863.210978] Lustre: lustre-OST0000: new disk, initializing [12863.214506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12863.228454] Lustre: 353798:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12864.939473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12864.961597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12865.062142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12869.855724] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12879.997334] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12884.922884] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12885.325106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12891.498987] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12891.695524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12895.718814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12895.737416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12895.737425] Lustre: Skipped 2 previous similar messages [12895.742773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12895.768780] Lustre: Skipped 1 previous similar message [12900.826791] Lustre: server umount lustre-OST0000 complete [12905.962755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12905.997790] Lustre: Skipped 1 previous similar message [12906.011525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12906.024207] Lustre: Skipped 1 previous similar message [12911.509318] Lustre: server umount lustre-MDT0000 complete [12915.763413] LustreError: 351823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614817 with bad export cookie 2370084857025058246 [12915.777585] LustreError: 351823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12915.792053] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12916.349961] Lustre: server umount lustre-MDT0001 complete [12925.972590] XFS (loop0): Unmounting Filesystem [12930.996415] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 23:13:51 (1789614831) [12938.790884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12939.383586] LustreError: 356357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12939.470422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12939.474331] Lustre: Skipped 2 previous similar messages [12944.394206] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12952.702598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12958.606925] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12968.097193] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12973.401960] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12980.492729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12988.733753] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [12998.692826] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13008.865629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13008.870924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13008.874154] Lustre: Skipped 1 previous similar message [13008.877141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13008.879371] LustreError: Skipped 1 previous similar message [13008.893767] Lustre: Skipped 3 previous similar messages [13013.472740] Lustre: server umount lustre-OST0000 complete [13024.930394] Lustre: server umount lustre-MDT0000 complete [13029.293546] LustreError: 356340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614930 with bad export cookie 2370084857025059128 [13029.304518] LustreError: 356340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13029.305360] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13029.840143] Lustre: server umount lustre-MDT0001 complete [13041.497655] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [13044.416747] Key type lgssc unregistered [13044.763502] LNet: 359985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13044.766226] LNetError: 359985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13044.787713] LNet: Removed LNI 192.168.201.105@tcp [13046.101203] Key type .llcrypt unregistered [13046.103152] Key type ._llcrypt unregistered [13077.099443] Key type ._llcrypt registered [13077.104480] Key type .llcrypt registered [13077.305954] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [13078.566906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13078.651111] alg: No test for adler32 (adler32-zlib) [13079.722974] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [13079.959639] LNet: Added LNI 192.168.201.105@tcp [8/256/0/180] [13081.655250] Key type lgssc registered [13083.000888] Lustre: Echo OBD driver; http://www.lustre.org/ [13092.206720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13092.232502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13093.826944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13099.015187] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13104.097747] LustreError: 361549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13104.119949] LustreError: 361549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13109.217938] LustreError: 361548:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13109.323318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13109.967842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13115.202823] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13124.422981] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13131.860511] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13140.290601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13140.832996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13148.867679] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13157.882523] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13176.804085] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13176.818663] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13176.847901] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13178.934102] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [13179.877155] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13184.013649] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [13184.021622] Lustre: Skipped 2 previous similar messages [13189.117338] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [13189.124395] Lustre: Skipped 2 previous similar messages [13190.111235] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13190.281434] Lustre: server umount lustre-MDT0001 complete [13194.262143] LustreError: 363295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13199.359668] LustreError: 363840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13199.419129] LustreError: 363840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13204.480176] LustreError: 363295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13204.492488] LustreError: 363295:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13204.801370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13205.044947] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13205.051157] Lustre: Found index 1 for lustre-MDT0001, updating log [13205.141321] Lustre: 361536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13205.474395] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13205.477448] Lustre: Skipped 2 previous similar messages [13205.658451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13210.621276] LustreError: 364967:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13210.631950] LustreError: 364967:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13210.648292] LustreError: 361540:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13210.750523] LustreError: 364972:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -17 [13210.766651] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13210.766651] [13211.151478] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13211.692739] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13226.464744] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13226.488894] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13226.495043] Lustre: Skipped 1 previous similar message [13232.838251] Lustre: server umount lustre-MDT0001 complete [13235.203787] LustreError: 361544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13245.521457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13245.696169] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13245.701630] Lustre: Found index 1 for lustre-MDT0001, updating log [13245.775647] Lustre: 361537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13246.429679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13251.578398] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13251.617623] LustreError: 366041:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13251.627153] LustreError: 366041:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13251.641420] LustreError: 361540:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13251.653261] LustreError: 361540:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13251.721900] LustreError: 366046:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13251.733794] LustreError: 366046:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13251.739405] LustreError: 366046:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -17 [13251.749891] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.105@tcp [13251.749891] [13252.697347] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13562.938522] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.5@tcp (stopping) [13562.957581] Lustre: Skipped 3 previous similar messages [13563.874970] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13563.881270] Lustre: Skipped 2 previous similar messages [13566.139753] Lustre: server umount lustre-MDT0001 complete [13568.003289] LustreError: 361544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13568.021205] LustreError: 361544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [13579.136953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13579.354668] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13579.365485] Lustre: Found index 1 for lustre-MDT0001, updating log [13579.446653] Lustre: 361537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13579.759979] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13584.884341] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13585.324880] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [13585.912937] LustreError: 374197:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13585.920325] LustreError: 374197:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13585.939904] LustreError: 361540:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13585.946138] LustreError: 361540:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13586.003325] LustreError: 374202:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13586.034328] LustreError: 374202:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.105@tcp: cfg command failed: rc = -114 [13586.040818] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13586.040818] [13912.544291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13912.558665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13912.577657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13912.587564] Lustre: Skipped 4 previous similar messages [13923.295159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13925.441418] Lustre: server umount lustre-MDT0000 complete [13927.908880] LustreError: 361549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13927.924326] LustreError: 361549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [13928.520352] LustreError: 361530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789615829 with bad export cookie 8774034748135539953 [13928.522360] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13928.534664] LustreError: 361530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13933.028559] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13933.051560] Lustre: Skipped 2 previous similar messages [13934.904143] Lustre: server umount lustre-MDT0001 complete [13939.995300] Lustre: server umount lustre-OST0000 complete [13951.193408] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [13961.195264] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [14007.593747] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 23:31:47 (1789615907) [14015.514715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14015.899634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14015.949835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14016.094560] Lustre: lustre-MDT0000: new disk, initializing [14016.226135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14016.241539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14021.917761] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [14035.146934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14035.583192] Lustre: 384561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14035.711306] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14035.719263] Lustre: Skipped 1 previous similar message [14035.820835] Lustre: lustre-MDT0001: new disk, initializing [14036.027168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14036.117720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14036.132683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14042.398231] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [14056.551696] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14064.212685] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14073.237729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14073.535738] Lustre: lustre-OST0000: new disk, initializing [14073.540837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14073.550160] Lustre: 386534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14073.676690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14081.341394] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_default_debug -1 all [14083.655171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14083.665211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14083.789131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14091.927365] Lustre: DEBUG MARKER: oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14100.967533] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [14100.983253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14101.012350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14101.022991] Lustre: Skipped 7 previous similar messages [14102.576962] Lustre: server umount lustre-MDT0000 complete [14104.546915] LustreError: 384573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14104.579315] LustreError: 384573:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [14107.485760] LustreError: 385356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789616008 with bad export cookie 8774034748135541906 [14107.492405] LustreError: MGC192.168.201.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14107.513449] LustreError: 385356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14107.558238] LustreError: 385368:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [14108.056747] Lustre: server umount lustre-MDT0001 complete [14123.160113] Lustre: server umount lustre-OST0000 complete [14158.670084] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing set_hostid [14169.376223] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing load_modules_local [14204.499529] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13941 sec ========= 23:35:04 (1789616104) [14206.377831] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:35:06 (1789616106) === [14210.086769] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:35:10 (1789616110) === [14258.695343] Lustre: DEBUG MARKER: oleg105-server.virtnet: executing unload_modules_local [14262.166763] Key type lgssc unregistered [14262.917530] LNet: 392902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14262.937261] LNetError: 392902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14262.965253] LNet: Removed LNI 192.168.201.105@tcp [14264.595407] Key type .llcrypt unregistered [14264.604217] Key type ._llcrypt unregistered