[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 577242926 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002347] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007447] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010123] LSM: Security Framework initializing [ 0.011000] Yama: becoming mindful. [ 0.011033] SELinux: Initializing. [ 0.012063] *** VALIDATE selinux *** [ 0.020669] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025539] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027129] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029066] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031071] *** VALIDATE tmpfs *** [ 0.032456] *** VALIDATE proc *** [ 0.033268] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.036028] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042376] debug: unmapping init [mem 0xffffffff88059000-0xffffffff88060fff] [ 0.044223] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045691] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046019] ... version: 2 [ 0.047008] ... bit width: 48 [ 0.048007] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050009] ... max period: 00007fffffffffff [ 0.051008] ... fixed-purpose events: 3 [ 0.052007] ... event mask: 000000070000000f [ 0.053349] rcu: Hierarchical SRCU implementation. [ 0.055424] smp: Bringing up secondary CPUs ... [ 0.056484] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.121113] smp: Brought up 1 node, 4 CPUs [ 0.123064] smpboot: Max logical packages: 1 [ 0.124009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.192024] node 0 deferred pages initialised in 66ms [ 0.196189] devtmpfs: initialized [ 0.197221] x86/mm: Memory block size: 128MB [ 0.200260] gcov: version magic: 0x41383552 [ 0.203100] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.204081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.205267] pinctrl core: initialized pinctrl subsystem [ 0.207000] [ 0.243026] ************************************************************* [ 0.245013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251013] ** ** [ 0.258013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.260012] ** ** [ 0.263031] ** This means that this kernel is built to expose internal ** [ 0.265010] ** IOMMU data structures, which may compromise security on ** [ 0.268011] ** your system. ** [ 0.272013] ** ** [ 0.278014] ** If you see this message and you are not debugging the ** [ 0.281010] ** kernel, report this immediately to your vendor! ** [ 0.284074] ** ** [ 0.286011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.289012] ************************************************************* [ 0.291759] NET: Registered protocol family 16 [ 0.293550] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.296059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.299076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.303037] cpuidle: using governor menu [ 0.304807] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.307802] PCI: Using configuration type 1 for base access [ 0.312137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.438198] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.440020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.443052] cryptd: max_cpu_qlen set to 1000 [ 0.446263] ACPI: Added _OSI(Module Device) [ 0.447000] ACPI: Added _OSI(Processor Device) [ 0.447000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.449015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.455244] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.462509] ACPI: Interpreter enabled [ 0.464066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.466014] ACPI: Using IOAPIC for interrupt routing [ 0.468183] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.473528] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.486222] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.489038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.492019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.496089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.502752] acpiphp: Slot [2] registered [ 0.504149] acpiphp: Slot [5] registered [ 0.506273] acpiphp: Slot [6] registered [ 0.508248] acpiphp: Slot [7] registered [ 0.510213] acpiphp: Slot [8] registered [ 0.511169] acpiphp: Slot [9] registered [ 0.513125] acpiphp: Slot [10] registered [ 0.515099] acpiphp: Slot [3] registered [ 0.517105] acpiphp: Slot [4] registered [ 0.518084] acpiphp: Slot [11] registered [ 0.520083] acpiphp: Slot [12] registered [ 0.521093] acpiphp: Slot [13] registered [ 0.523096] acpiphp: Slot [14] registered [ 0.525168] acpiphp: Slot [15] registered [ 0.527120] acpiphp: Slot [16] registered [ 0.528153] acpiphp: Slot [17] registered [ 0.530200] acpiphp: Slot [18] registered [ 0.532149] acpiphp: Slot [19] registered [ 0.533190] acpiphp: Slot [20] registered [ 0.535104] acpiphp: Slot [21] registered [ 0.537125] acpiphp: Slot [22] registered [ 0.539117] acpiphp: Slot [23] registered [ 0.541103] acpiphp: Slot [24] registered [ 0.542172] acpiphp: Slot [25] registered [ 0.544155] acpiphp: Slot [26] registered [ 0.546122] acpiphp: Slot [27] registered [ 0.548118] acpiphp: Slot [28] registered [ 0.550145] acpiphp: Slot [29] registered [ 0.552197] acpiphp: Slot [30] registered [ 0.554103] acpiphp: Slot [31] registered [ 0.555055] PCI host bridge to bus 0000:00 [ 0.557020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.559020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.562022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.565022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.567023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.570096] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.572168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.575176] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.579428] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.586018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.592070] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.595018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.597014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.599016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.604166] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.606748] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.609044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.612223] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.617010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.630012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.633057] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.638536] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.649017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.659016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.006052] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 1.061659] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.075017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.084021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.105024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.117845] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.123029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.137025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.160024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.173941] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.212149] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.223022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.239021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.249426] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.257023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.264022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.284023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.295000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.302023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.309025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.324023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.339096] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.341635] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.344611] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.347707] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.350276] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.397047] iommu: Default domain type: Passthrough [ 1.398787] SCSI subsystem initialized [ 1.399230] ACPI: bus type USB registered [ 1.400193] usbcore: registered new interface driver usbfs [ 1.402117] usbcore: registered new interface driver hub [ 1.403099] usbcore: registered new device driver usb [ 1.405190] pps_core: LinuxPPS API ver. 1 registered [ 1.407072] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.410077] PTP clock support registered [ 1.414066] EDAC MC: Ver: 3.0.0 [ 1.416178] PCI: Using ACPI for IRQ routing [ 1.419073] NetLabel: Initializing [ 1.420011] NetLabel: domain hash size = 128 [ 1.421015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.423102] NetLabel: unlabeled traffic allowed by default [ 1.425148] vgaarb: loaded [ 1.427518] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.428013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.432947] clocksource: Switched to clocksource kvm-clock [ 1.586349] VFS: Disk quotas dquot_6.6.0 [ 1.587980] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.590637] *** VALIDATE ramfs *** [ 1.592094] *** VALIDATE hugetlbfs *** [ 1.594622] pnp: PnP ACPI init [ 1.597697] pnp: PnP ACPI: found 6 devices [ 1.729548] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.733464] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.736158] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.738813] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.741627] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.744620] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.748363] NET: Registered protocol family 2 [ 1.751373] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.757389] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.762053] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.768104] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.771666] TCP: Hash tables configured (established 65536 bind 65536) [ 1.775200] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.779309] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.783283] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.787244] NET: Registered protocol family 1 [ 1.805742] RPC: Registered named UNIX socket transport module. [ 1.816455] RPC: Registered udp transport module. [ 1.819372] RPC: Registered tcp transport module. [ 1.821586] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.826798] NET: Registered protocol family 44 [ 1.828673] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.831432] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.834429] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.837207] PCI: CLS 0 bytes, default 64 [ 1.839735] Unpacking initramfs... [ 3.560757] debug: unmapping init [mem 0xffff8ad43cc54000-0xffff8ad43ffbffff] [ 3.567641] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.569944] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.574176] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.175798] Initialise system trusted keyrings [ 4.177860] Key type blacklist registered [ 4.179746] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.189653] zbud: loaded [ 4.192127] *** VALIDATE nfs *** [ 4.192852] *** VALIDATE nfs4 *** [ 4.193963] pstore: using deflate compression [ 4.196837] Platform Keyring initialized [ 4.316644] NET: Registered protocol family 38 [ 4.318524] Key type asymmetric registered [ 4.320819] Asymmetric key parser 'x509' registered [ 4.322238] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.325966] io scheduler mq-deadline registered [ 4.327400] io scheduler kyber registered [ 4.332971] io scheduler bfq registered [ 4.334779] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.337661] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.340573] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.344549] ACPI: Power Button [PWRF] [ 4.350200] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.357934] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.374314] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.391537] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.515151] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.556964] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.592055] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.599398] Non-volatile memory driver v1.3 [ 4.601200] Linux agpgart interface v0.103 [ 4.655738] virtio_blk virtio1: [vda] 146944 512-byte logical blocks (75.2 MB/71.8 MiB) [ 4.660292] vda: detected capacity change from 0 to 75235328 [ 4.677248] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.682210] vdb: detected capacity change from 0 to 1073741824 [ 4.701325] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.706453] vdc: detected capacity change from 0 to 2621440000 [ 4.724113] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.727933] vdd: detected capacity change from 0 to 2621440000 [ 4.751718] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.760360] vde: detected capacity change from 0 to 4294967296 [ 4.804713] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.813713] vdf: detected capacity change from 0 to 4294967296 [ 4.834203] libphy: Fixed MDIO Bus: probed [ 4.842464] usbcore: registered new interface driver usbserial_generic [ 4.845245] usbserial: USB Serial support registered for generic [ 4.847635] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.853172] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.855124] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.857770] mousedev: PS/2 mouse device common for all mice [ 4.861263] rtc_cmos 00:05: RTC can wake from S4 [ 4.863207] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.867949] rtc_cmos 00:05: registered as rtc0 [ 4.868056] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.868095] intel_pstate: CPU model not supported [ 4.875326] hid: raw HID events driver (C) Jiri Kosina [ 4.883166] usbcore: registered new interface driver usbhid [ 4.885819] usbhid: USB HID core driver [ 4.888269] drop_monitor: Initializing network drop monitor service [ 4.889408] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.891759] Initializing XFRM netlink socket [ 4.897943] NET: Registered protocol family 10 [ 4.900373] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.905024] Segment Routing with IPv6 [ 4.908349] NET: Registered protocol family 17 [ 4.910853] mpls_gso: MPLS GSO support [ 4.921949] RAS: Correctable Errors collector initialized. [ 4.924964] AVX version of gcm_enc/dec engaged. [ 4.927185] AES CTR mode by8 optimization enabled [ 5.065269] sched_clock: Marking stable (5065182043, 0)->(6214366282, -1149184239) [ 5.069264] registered taskstats version 1 [ 5.072194] Loading compiled-in X.509 certificates [ 5.074479] zswap: loaded using pool lzo/zbud [ 5.116841] Key type big_key registered [ 5.138233] Key type encrypted registered [ 5.139869] ima: No TPM chip found, activating TPM-bypass! [ 5.142188] ima: Allocated hash algorithm: sha1 [ 5.144197] ima: No architecture policies found [ 5.146424] evm: Initialising EVM extended attributes: [ 5.148859] evm: security.selinux [ 5.150506] evm: security.ima [ 5.151599] evm: security.capability [ 5.153261] evm: HMAC attrs: 0x1 [ 5.156593] rtc_cmos 00:05: setting system clock to 2026-08-27 14:32:04 UTC (1787841124) [ 5.164487] debug: unmapping init [mem 0xffffffff89003000-0xffffffff891fffff] [ 5.168457] debug: unmapping init [mem 0xffffffff87d82000-0xffffffff88058fff] [ 5.176227] Write protecting the kernel read-only data: 28672k [ 5.179473] debug: unmapping init [mem 0xffffffff86403000-0xffffffff865fffff] [ 5.183162] debug: unmapping init [mem 0xffffffff86d14000-0xffffffff86dfffff] [ 5.234665] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.245507] systemd[1]: Detected virtualization kvm. [ 5.247743] systemd[1]: Detected architecture x86-64. [ 5.250301] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.281873] systemd[1]: No hostname configured. [ 5.284322] systemd[1]: Set hostname to . [ 5.286969] random: systemd: uninitialized urandom read (16 bytes read) [ 5.290195] systemd[1]: Initializing machine ID from random generator. [ 5.390643] random: ln: uninitialized urandom read (6 bytes read) [ 5.634361] random: systemd: uninitialized urandom read (16 bytes read) [ 5.637973] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.652421] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.660650] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.585607] device-mapper: uevent: version 1.0.3 [ 6.595149] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 8.142383] virtio_net virtio0 ens2: renamed from eth0 [ 8.154067] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.661180] scsi host0: ata_piix [ 8.668708] scsi host1: ata_piix [ 8.669956] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.673056] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.665424] random: crng init done [ 13.669883] random: 7 urandom warning(s) missed due to ratelimiting [ 15.412441] dracut-initqueue[576]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.938297] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.187975] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.825817] SELinux: Disabled at runtime. [ 20.905446] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.916196] systemd[1]: Detected virtualization kvm. [ 20.918153] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.515728] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.521896] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.528624] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.533086] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.536946] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.546320] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.555337] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK [0[ 22.936538] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 24.955692] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ 25.274678] hrtimer: interrupt took 4979847 ns [ OK ] Started udev Kernel Device Manager. [ 26.961132] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.975588] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.622079] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.773527] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 34.987968] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit)[ 36.164963] NFS: Registering the id_resolver key type [ 36.172498] Key type id_resolver registered [ 36.179230] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg106-server login: [ 111.493686] libcfs: loading out-of-tree module taints kernel. [ 111.525368] Key type ._llcrypt registered [ 111.527305] Key type .llcrypt registered [ 111.631242] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 133.797353] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 136.288095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 136.297319] alg: No test for adler32 (adler32-zlib) [ 137.695426] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 138.489272] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 140.311579] Key type lgssc registered [ 141.911535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 162.651244] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 210.214903] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 223.879121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 223.901640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.256695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 225.279254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 225.400792] Lustre: lustre-MDT0000: new disk, initializing [ 225.522865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.543687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 230.261106] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.289928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 246.450711] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 246.491399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 246.499245] Lustre: Skipped 1 previous similar message [ 246.646132] Lustre: lustre-MDT0001: new disk, initializing [ 246.812454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.871045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 246.878796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 252.459980] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.444258] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 267.550324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 267.876724] Lustre: lustre-OST0000: new disk, initializing [ 267.881616] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 267.889850] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 267.984487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.649903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 270.684404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 270.830469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 275.121709] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 290.445408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 290.578678] Lustre: lustre-OST0001: new disk, initializing [ 290.587267] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 290.593183] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 290.671514] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 297.273102] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 299.541094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 299.554450] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 299.639568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 311.208973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 320.526630] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 330.111044] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing check_logdir /tmp/testlogs/ [ 336.742678] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing yml_node [ 342.306277] Lustre: DEBUG MARKER: Client: 2.17.57.96 [ 344.845108] Lustre: DEBUG MARKER: MDS: 2.17.57.96 [ 349.801691] Lustre: DEBUG MARKER: OSS: 2.17.57.96 [ 353.127907] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:37:50 EDT 2026 [ 377.046757] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 378.481260] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 395.237126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 395.255542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 395.274667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 396.768980] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.769842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 396.830985] Lustre: Skipped 3 previous similar messages [ 402.291792] Lustre: server umount lustre-MDT0000 complete [ 406.772103] LustreError: 8443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841526 with bad export cookie 4337365035686535000 [ 406.794238] LustreError: 8443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 406.806113] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 407.010453] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.011592] LustreError: 6514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.027072] Lustre: Skipped 2 previous similar messages [ 407.031472] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 407.038314] LustreError: 6514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 407.053418] Lustre: Skipped 1 previous similar message [ 411.040930] LustreError: 11079:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.041591] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 411.091850] LustreError: 11079:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 411.100146] Lustre: Skipped 1 previous similar message [ 416.233165] LustreError: 6513:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 416.233284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 416.260996] LustreError: 6513:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 416.279892] Lustre: Skipped 1 previous similar message [ 421.343300] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 421.924549] Lustre: server umount lustre-MDT0001 complete [ 430.848718] Lustre: server umount lustre-OST0000 complete [ 435.960517] Lustre: server umount lustre-OST0001 complete [ 445.585767] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 455.748787] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 496.181095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.415713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 496.459050] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 496.553972] Lustre: lustre-MDT0000: new disk, initializing [ 496.643914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.673919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 501.789355] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 511.635705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.769728] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 511.842932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 511.847760] Lustre: Skipped 1 previous similar message [ 511.926567] Lustre: lustre-MDT0001: new disk, initializing [ 512.036870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 512.121075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 512.165102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 517.258511] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 529.373147] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 537.554529] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 545.427152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.790417] Lustre: lustre-OST0000: new disk, initializing [ 545.795394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 545.807827] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 545.922926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.226689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 547.238543] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 547.420090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 553.947303] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 566.366217] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 571.948831] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 572.179637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 577.137258] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 577.314625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 583.147930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.158366] Lustre: Skipped 1 previous similar message [ 583.167556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.175177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 583.197608] Lustre: Skipped 3 previous similar messages [ 586.073745] Lustre: server umount lustre-OST0000 complete [ 593.387704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.421331] Lustre: Skipped 2 previous similar messages [ 598.110303] Lustre: server umount lustre-MDT0000 complete [ 598.511824] LustreError: 15087:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.540138] LustreError: 15087:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 603.616726] LustreError: 15086:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.637989] LustreError: 15086:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 603.833152] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841723 with bad export cookie 4337365035686536533 [ 603.836313] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.844503] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 604.304422] Lustre: server umount lustre-MDT0001 complete [ 610.825331] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 10:42:08 (1787841728) [ 620.805888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.588996] LustreError: 19310:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.813973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 627.726639] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 638.347193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.662897] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 651.122191] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 657.298548] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 664.710125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 671.645702] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 679.552875] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 686.470581] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 694.760062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.770397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 694.781682] Lustre: Skipped 1 previous similar message [ 694.786611] Lustre: Skipped 2 previous similar messages [ 699.445271] Lustre: server umount lustre-MDT0000 complete [ 699.877940] LustreError: 20039:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.920950] LustreError: 20039:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 704.018830] LustreError: 19291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841823 with bad export cookie 4337365035686537415 [ 704.043534] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.454966] Lustre: server umount lustre-MDT0001 complete [ 719.322760] Lustre: server umount lustre-OST0000 complete [ 751.917993] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 763.384601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.226282] LustreError: 23715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 764.356145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.370754] Lustre: Skipped 2 previous similar messages [ 769.973270] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 779.923660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 786.333513] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 789.941654] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 798.923132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.040152] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 817.727747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 817.886482] Lustre: lustre-OST0001: new disk, initializing [ 817.891377] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 817.898032] Lustre: 26030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 819.509121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 819.524261] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 819.632827] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 825.685808] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 836.889187] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 840.656876] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 855.542107] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 855.559168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.575749] Lustre: Skipped 4 previous similar messages [ 855.603504] Lustre: Skipped 3 previous similar messages [ 860.657638] LustreError: 25842:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.669224] LustreError: 25842:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 860.733293] Lustre: server umount lustre-MDT0000 complete [ 865.731653] LustreError: 23697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841985 with bad export cookie 4337365035686538262 [ 865.744600] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 865.749251] LustreError: 23697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 866.173899] Lustre: server umount lustre-MDT0001 complete [ 873.173357] Lustre: server umount lustre-OST0000 complete [ 877.530947] Lustre: server umount lustre-OST0001 complete [ 894.667217] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 905.246751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 905.789695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.794443] Lustre: Skipped 3 previous similar messages [ 910.705833] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 919.213041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.177944] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 927.504153] Lustre: 30199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 935.305502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 942.502970] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 952.502091] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 960.343260] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 968.879220] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 973.020447] Lustre: 32076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 983.528621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.537491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.545924] Lustre: Skipped 3 previous similar messages [ 983.555560] Lustre: Skipped 5 previous similar messages [ 988.221548] Lustre: server umount lustre-MDT0000 complete [ 988.664490] LustreError: 29051:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.700534] LustreError: 29051:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 992.225350] LustreError: 31390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842111 with bad export cookie 4337365035686539557 [ 992.234035] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 992.249270] LustreError: 31390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 992.775204] Lustre: server umount lustre-MDT0001 complete [ 1007.654039] Lustre: server umount lustre-OST0000 complete [ 1010.658798] Lustre: 3639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842113/real 1787842113] req@ffff8ad383de83c0 x1874687434214144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842129 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1011.625544] Lustre: server umount lustre-OST0001 complete [ 1021.397099] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 10:48:59 (1787842139) [ 1054.803373] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1066.633471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.111577] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1081.070372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1087.072863] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1090.328973] Lustre: 36031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1097.824763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.964267] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1116.380370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.972565] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1132.543526] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1136.383221] Lustre: 37909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1142.245080] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.255117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.275896] Lustre: Skipped 5 previous similar messages [ 1142.301905] Lustre: Skipped 3 previous similar messages [ 1148.092831] Lustre: server umount lustre-MDT0000 complete [ 1151.569871] LustreError: 34863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842270 with bad export cookie 4337365035686540656 [ 1151.579774] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.963195] Lustre: server umount lustre-MDT0001 complete [ 1165.910087] Lustre: server umount lustre-OST0000 complete [ 1167.839189] Lustre: 3637:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842271/real 1787842271] req@ffff8ad4a78d0000 x1874687434264832/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1171.051803] Lustre: server umount lustre-OST0001 complete [ 1181.577171] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 10:51:40 (1787842300) [ 1198.083443] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1209.133714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.600645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.605467] Lustre: Skipped 7 previous similar messages [ 1214.410922] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1223.378657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.712424] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1231.664405] Lustre: 41202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1239.004179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1246.319747] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1248.680783] LustreError: 41556:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.705431] LustreError: 41556:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 1256.072614] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1262.767802] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1271.371919] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1275.243241] Lustre: 43079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1276.509432] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1276.523417] Lustre: Skipped 1 previous similar message [ 1279.971851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1279.984975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.012527] Lustre: Skipped 2 previous similar messages [ 1282.588446] Lustre: server umount lustre-MDT0000 complete [ 1287.236535] LustreError: 40034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842406 with bad export cookie 4337365035686541769 [ 1287.240921] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.243568] LustreError: 40034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1293.815954] Lustre: server umount lustre-MDT0001 complete [ 1304.234650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.723803] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3c316a85054bbe0d [ 1311.744812] Lustre: MGC192.168.201.106@tcp: Connection restored to 0@lo (at 0@lo) [ 1317.978152] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1322.476133] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1328.435557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.399897] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1332.402753] Lustre: Skipped 1 previous similar message [ 1334.741748] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1342.971146] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1345.166759] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1351.862698] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1353.584739] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1359.841827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1365.944856] Lustre: server umount lustre-MDT0000 complete [ 1370.245347] LustreError: 40033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842489 with bad export cookie 4337365035686542861 [ 1370.268060] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1370.271708] LustreError: 40033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1370.712761] Lustre: server umount lustre-MDT0001 complete [ 1385.185278] Lustre: server umount lustre-OST0000 complete [ 1399.864755] Lustre: server umount lustre-OST0001 complete [ 1409.800064] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 10:55:28 (1787842528) [ 1429.085502] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1439.939110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.731278] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1454.948661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.584127] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1462.447411] Lustre: 48842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1469.082878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1475.784948] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1484.387475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.853937] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1499.039294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1503.218330] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1516.519764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1516.527874] Lustre: Skipped 21 previous similar messages [ 1517.168663] Lustre: server umount lustre-MDT0000 complete [ 1520.750318] LustreError: 47672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842640 with bad export cookie 4337365035686543687 [ 1520.763083] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1520.766923] LustreError: 47672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1521.299151] Lustre: server umount lustre-MDT0001 complete [ 1535.734574] Lustre: server umount lustre-OST0000 complete [ 1537.512609] Lustre: 3639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842640/real 1787842640] req@ffff8ad48375a1c0 x1874687434407936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787842656 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1537.557731] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1537.581593] Lustre: Skipped 13 previous similar messages [ 1540.393547] Lustre: server umount lustre-OST0001 complete [ 1551.067873] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 10:57:49 (1787842669) [ 1553.855287] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1555.954740] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 10:57:54 (1787842674) [ 1567.153266] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 1576.772351] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1600.310174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.681541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1600.770896] Lustre: lustre-MDT0000: new disk, initializing [ 1600.924444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1605.186480] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1615.405203] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1621.956342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1622.203525] Lustre: lustre-OST0000: new disk, initializing [ 1622.207258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1622.213058] Lustre: Skipped 1 previous similar message [ 1622.217566] Lustre: 54915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1623.748189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1623.758562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1623.802273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1630.173607] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1639.805851] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1643.087465] Lustre: DEBUG MARKER: create test files [ 1675.235202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1677.280749] Lustre: server umount lustre-OST0000 complete [ 1680.916491] Lustre: server umount lustre-MDT0000 complete [ 1693.314458] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1696.105217] Key type lgssc unregistered [ 1696.478985] LNet: 56551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.488813] LNetError: 56551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.503505] LNet: Removed LNI 192.168.201.106@tcp [ 1697.624196] Key type .llcrypt unregistered [ 1697.629825] Key type ._llcrypt unregistered [ 1726.383778] Key type ._llcrypt registered [ 1726.385652] Key type .llcrypt registered [ 1726.490920] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1728.794522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.836652] alg: No test for adler32 (adler32-zlib) [ 1729.825726] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 1730.031566] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1731.711156] Key type lgssc registered [ 1733.008836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1739.735077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1739.771973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.345175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.536608] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1754.013195] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1761.459758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1761.726630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.807866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1765.937374] Lustre: 59297:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1765.947742] Lustre: 59297:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1765.953232] Lustre: 59297:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1765.963228] Lustre: 59297:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1765.976205] Lustre: 59297:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1765.984786] Lustre: 59297:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1767.989528] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1775.060869] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1780.506730] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1785.595782] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1794.541951] Lustre: server umount lustre-OST0000 complete [ 1808.533591] Lustre: server umount lustre-MDT0000 complete [ 1820.082151] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1823.375481] Key type lgssc unregistered [ 1823.686336] LNet: 60909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.695699] LNetError: 60909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.723845] LNet: Removed LNI 192.168.201.106@tcp [ 1824.698950] Key type .llcrypt unregistered [ 1824.706154] Key type ._llcrypt unregistered [ 1859.711354] Key type ._llcrypt registered [ 1859.719491] Key type .llcrypt registered [ 1859.840666] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1861.057064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1861.301733] alg: No test for adler32 (adler32-zlib) [ 1862.475121] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 1862.880648] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1864.712738] Key type lgssc registered [ 1866.283634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1875.976757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1876.006296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.712442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.089960] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1892.884604] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1901.396291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1902.007676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1903.043181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1903.198863] Lustre: 63744:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1903.207260] Lustre: 63744:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1903.216419] Lustre: 63744:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1903.223574] Lustre: 63744:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1903.232874] Lustre: 63744:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1903.243286] Lustre: 63744:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1909.959774] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1918.077952] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1924.748259] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1933.793851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.832078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1938.912674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.817357] Lustre: server umount lustre-OST0000 complete [ 1944.077454] Lustre: server umount lustre-MDT0000 complete [ 1954.967552] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1958.052842] Key type lgssc unregistered [ 1958.281788] LNet: 65307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.288831] LNetError: 65307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.300731] LNet: Removed LNI 192.168.201.106@tcp [ 1959.028544] Key type .llcrypt unregistered [ 1959.030739] Key type ._llcrypt unregistered [ 1979.511419] Key type ._llcrypt registered [ 1979.514748] Key type .llcrypt registered [ 1979.633974] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 1998.739519] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2000.354917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2000.380573] alg: No test for adler32 (adler32-zlib) [ 2001.536160] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 2001.770521] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2003.471207] Key type lgssc registered [ 2004.616592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2046.357928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2046.422605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.725734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2047.785796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2047.905188] Lustre: lustre-MDT0000: new disk, initializing [ 2047.984945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.005653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2052.657371] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2065.853298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.977787] Lustre: 68463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2066.047362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2066.050964] Lustre: Skipped 1 previous similar message [ 2066.121870] Lustre: lustre-MDT0001: new disk, initializing [ 2066.230530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2066.263035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2066.272367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2070.828048] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2080.548552] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2086.545462] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.297357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.619676] Lustre: lustre-OST0000: new disk, initializing [ 2093.623664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2093.633549] Lustre: 70432:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2093.738369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.365245] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2102.880095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2102.888360] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2102.977514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2111.570063] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2117.248686] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2117.627658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2121.490353] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2121.691533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2123.746867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.748087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2123.772154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2123.781700] Lustre: Skipped 1 previous similar message [ 2128.867954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2128.881607] Lustre: Skipped 1 previous similar message [ 2129.424798] Lustre: server umount lustre-OST0000 complete [ 2133.987204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.002176] Lustre: Skipped 1 previous similar message [ 2134.009948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.106194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.112647] Lustre: Skipped 2 previous similar messages [ 2144.224976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2144.238099] Lustre: Skipped 1 previous similar message [ 2148.319157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2148.655327] Lustre: server umount lustre-MDT0000 complete [ 2152.841491] LustreError: 69268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843272 with bad export cookie 12674428006465676663 [ 2152.848267] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2152.854771] LustreError: 69268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2153.156291] Lustre: server umount lustre-MDT0001 complete [ 2163.505878] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 11:08:02 (1787843282) [ 2173.252808] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2211.748897] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 2224.491762] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2265.067807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.396195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2265.435291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2265.560452] Lustre: lustre-MDT0000: new disk, initializing [ 2265.732687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.763040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2270.308859] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2281.102964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2281.287338] Lustre: 76453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2281.329132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2281.332791] Lustre: Skipped 1 previous similar message [ 2281.393608] Lustre: lustre-MDT0001: new disk, initializing [ 2281.482831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2281.513553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2281.528216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2286.430196] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2296.551898] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2302.750882] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2310.372115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2310.608775] Lustre: lustre-OST0000: new disk, initializing [ 2310.612493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2310.617564] Lustre: 78425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2310.713227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.299461] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2319.417816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2319.443293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2319.539167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2327.444235] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2333.126316] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2333.519181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2339.675583] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2339.937902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2345.442892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2345.443737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.456717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2345.486335] Lustre: Skipped 2 previous similar messages [ 2348.620570] Lustre: server umount lustre-OST0000 complete [ 2353.122412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2353.132758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2359.203680] Lustre: server umount lustre-MDT0000 complete [ 2360.801070] LustreError: 76465:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.813237] LustreError: 76465:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2363.623463] LustreError: 76445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843482 with bad export cookie 12674428006465677538 [ 2363.633037] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2363.641520] LustreError: 76445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2364.147353] Lustre: server umount lustre-MDT0001 complete [ 2373.623463] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 11:11:32 (1787843492) [ 2380.652292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2381.295356] LustreError: 80838:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.424592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.405838] LustreError: 80839:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2386.989802] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2391.522507] LustreError: 80838:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.648085] LustreError: 80839:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.920420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.941651] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2410.911992] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2417.637235] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2426.906379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2427.683189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2427.688512] Lustre: Skipped 1 previous similar message [ 2434.105289] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2441.730209] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2446.103158] Lustre: 83510:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 181e7a85-d857-4725-a0ce-b7d51ff9a6b0 at adminstrative request [ 2448.382237] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.401889] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2458.661069] Lustre: MGS: Client 181e7a85-d857-4725-a0ce-b7d51ff9a6b0 (at 0@lo) reconnecting [ 2458.680373] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xafe49d1eb0a62c5b to 0xafe49d1eb0a62ef4 [ 2458.708563] Lustre: MGC192.168.201.106@tcp: Connection restored to 0@lo (at 0@lo) [ 2485.518400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2485.809121] Lustre: lustre-OST0001: new disk, initializing [ 2485.815610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2485.823231] Lustre: 83903:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2492.072472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2492.084418] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2492.251922] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2493.522921] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2503.878371] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2507.751732] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.755589] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2507.767455] Lustre: Skipped 1 previous similar message [ 2507.772253] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2507.792972] Lustre: Skipped 5 previous similar messages [ 2513.394056] Lustre: server umount lustre-OST0001 complete [ 2517.992931] LustreError: 82661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2518.001037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2518.002055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2518.020360] LustreError: Skipped 1 previous similar message [ 2518.031924] Lustre: Skipped 2 previous similar messages [ 2532.321186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2532.488864] Lustre: server umount lustre-OST0000 complete [ 2536.418313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2536.428064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.594767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.614636] Lustre: Skipped 12 previous similar messages [ 2550.239194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2550.456541] Lustre: server umount lustre-MDT0000 complete [ 2553.832380] LustreError: 80839:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.850269] LustreError: 80839:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 2553.897744] LustreError: 80820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843673 with bad export cookie 12674428006465679092 [ 2553.900682] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.909203] LustreError: 80820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2554.237409] Lustre: server umount lustre-MDT0001 complete [ 2563.985927] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 11:14:42 (1787843682) [ 2565.983198] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2567.700810] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 11:14:46 (1787843686) [ 2569.157968] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2571.540060] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 11:14:49 (1787843689) [ 2573.300614] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2575.042253] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 11:14:53 (1787843693) [ 2576.817846] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2578.424798] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 11:14:57 (1787843697) [ 2580.001328] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2587.356598] loop: module loaded [ 2587.372270] loop0: detected capacity change from 0 to 4294967296 [ 2587.455851] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2589.602433] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2593.611502] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2598.100711] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2625.269083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2625.334966] LustreError: 88031:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2625.342008] LustreError: 88031:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2625.348696] LustreError: 88031:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2625.352960] LustreError: 88031:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2646.439371] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 2655.885467] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2690.341209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.677800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2690.708382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2690.771670] Lustre: lustre-MDT0000: new disk, initializing [ 2690.843718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.848691] Lustre: Skipped 1 previous similar message [ 2690.860958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2695.226327] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2705.041181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.148576] Lustre: 90897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2705.376539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2709.717784] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2718.498401] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2725.000216] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2732.931873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2733.293159] Lustre: 92864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2740.855371] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2742.864155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2742.991586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2751.743703] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2756.518248] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2756.731369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2761.477607] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2761.772110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2763.746656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2763.751681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.764328] LustreError: Skipped 1 previous similar message [ 2763.765672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2763.765688] Lustre: Skipped 2 previous similar messages [ 2763.826247] Lustre: Skipped 2 previous similar messages [ 2769.853768] Lustre: server umount lustre-OST0000 complete [ 2780.038313] Lustre: server umount lustre-MDT0000 complete [ 2783.760425] LustreError: 91702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843903 with bad export cookie 12674428006465679708 [ 2783.771149] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2783.790524] LustreError: 91702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2784.152319] Lustre: server umount lustre-MDT0001 complete [ 2792.620446] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 11:18:31 (1787843911) [ 2800.014834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.584696] LustreError: 95276:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.762368] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2814.998464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.133519] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2830.698933] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2838.913042] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2846.477284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2846.702868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2846.715691] Lustre: Skipped 4 previous similar messages [ 2854.212221] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2863.477581] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2869.452431] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2870.857193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2876.487191] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2876.804654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2884.370777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2884.672130] Lustre: lustre-OST0001: new disk, initializing [ 2884.677415] Lustre: Skipped 2 previous similar messages [ 2884.687926] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2884.705387] Lustre: Skipped 3 previous similar messages [ 2884.715063] Lustre: 98654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2891.758947] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2894.509055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2894.536457] Lustre: Skipped 2 previous similar messages [ 2894.548863] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2894.694581] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2902.155936] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2907.086719] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2907.349563] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2912.549082] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2912.813022] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2933.783919] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2958.859143] Lustre: Failing over lustre-MDT0000 [ 2959.460440] Lustre: server umount lustre-MDT0000 complete [ 2961.172026] LustreError: 95272:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.183723] LustreError: 95272:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2961.393268] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.420261] Lustre: Skipped 4 previous similar messages [ 2976.735732] Lustre: 66687:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844080/real 1787844080] req@0000000045490a8b x1874689388727808/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787844096 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2976.772283] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2986.473449] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2986.479761] LDISKFS-fs (dm-0): recovery complete [ 2986.497192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.986923] LustreError: 66684:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004887b260 x1874689388736384/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2987.332869] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2991.872639] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2991.872985] Lustre: 97117:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2992.344995] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2992.609553] Lustre: 95276:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2992.623770] Lustre: 95276:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 1 previous similar message [ 2992.641655] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2992.745856] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2993.072920] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2993.402042] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2993.426460] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 1 previous similar message [ 2993.759498] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2993.770022] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 1 previous similar message [ 2993.796315] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2993.807400] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 7 previous similar messages [ 2994.463380] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2994.478588] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 2 previous similar messages [ 2994.830658] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2994.840427] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 2 previous similar messages [ 2995.800287] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2995.815515] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 5 previous similar messages [ 2996.464234] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2996.486755] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 5 previous similar messages [ 2997.119145] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2997.125401] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 6 previous similar messages [ 2999.958119] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2999.974372] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 12 previous similar messages [ 3000.609726] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3000.629731] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 12 previous similar messages [ 3001.263156] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3001.280115] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 12 previous similar messages [ 3007.966672] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3007.995594] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 24 previous similar messages [ 3008.664366] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3008.684321] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 24 previous similar messages [ 3009.327151] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3009.330348] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 24 previous similar messages [ 3023.972139] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3023.986678] Lustre: 101504:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 49 previous similar messages [ 3024.977778] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3024.994805] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 50 previous similar messages [ 3025.631184] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3025.638110] LustreError: 101504:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 50 previous similar messages [ 3051.071155] Lustre: 101504:0:(ldlm_lib.c:2457:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3051.078726] Lustre: 101504:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3051.088868] Lustre: 101504:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 55269fbc-8cfe-432e-a4d7-ac5c8ec55cc3@192.168.201.6@tcp [ 3051.101231] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3051.107672] Lustre: 101504:0:(ldlm_lib.c:1977:abort_req_replay_queue()) @@@ aborted: req@000000000ffdd38c x1874689369643520/t0(8589934779) o101->55269fbc-8cfe-432e-a4d7-ac5c8ec55cc3@192.168.201.6@tcp:406/0 lens 592/0 e 0 to 0 dl 1787844181 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3051.166177] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3051.179111] Lustre: lustre-MDT0000: Denying connection for new client 55269fbc-8cfe-432e-a4d7-ac5c8ec55cc3 (at 192.168.201.6@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3051.195129] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3051.246031] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3051.263683] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3051.278179] Lustre: Skipped 2 previous similar messages [ 3051.315649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 3051.320127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 3072.481966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3072.500901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3072.509244] Lustre: Skipped 7 previous similar messages [ 3076.556878] Lustre: server umount lustre-OST0000 complete [ 3087.068767] Lustre: server umount lustre-OST0001 complete [ 3092.448411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.458894] Lustre: Skipped 5 previous similar messages [ 3098.252576] Lustre: server umount lustre-MDT0000 complete [ 3100.133830] LustreError: 95276:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.146806] LustreError: 95276:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 45 previous similar messages [ 3102.066311] LustreError: 95256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844221 with bad export cookie 12674428006465730934 [ 3102.082765] LustreError: 95256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3102.087507] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3102.509211] Lustre: server umount lustre-MDT0001 complete [ 3112.877607] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 11:23:51 (1787844231) [ 3121.789812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3121.797782] Lustre: *** cfs_fail_loc=197, val=0*** [ 3137.311617] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3142.431677] LustreError: 103487:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.106@tcp: failed processing log, type 4: rc = -110 [ 3147.601726] Lustre: *** cfs_fail_loc=197, val=0*** [ 3147.608965] Lustre: Skipped 65 previous similar messages [ 3168.159487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.167205] Lustre: Skipped 2 previous similar messages [ 3175.299629] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3183.282443] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3185.897878] Lustre: server umount lustre-OST0000 complete [ 3194.691831] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 11:25:13 (1787844313) [ 3216.866725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3222.289848] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3232.779632] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3236.979445] LustreError: 105139:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3236.992548] LustreError: 105139:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3237.013880] LustreError: 105139:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3237.527669] Lustre: server umount lustre-MDT0000 complete [ 3275.632216] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 3289.039154] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3327.389106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.628659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3327.662473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3327.750923] Lustre: lustre-MDT0000: new disk, initializing [ 3327.862518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3332.271244] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3342.670729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.813524] Lustre: 109315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3343.090956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3347.772360] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3359.000975] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3365.044470] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3372.080491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.448183] Lustre: 111284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3379.961101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3380.450671] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3391.835227] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3398.245756] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3398.422834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3404.843628] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3405.162513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3410.920532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.926815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3410.938244] Lustre: Skipped 1 previous similar message [ 3410.945386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3410.945397] Lustre: Skipped 8 previous similar messages [ 3410.957056] LustreError: Skipped 3 previous similar messages [ 3422.175513] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3422.427511] Lustre: server umount lustre-OST0000 complete [ 3433.631737] Lustre: server umount lustre-MDT0000 complete [ 3436.513217] LustreError: 109324:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.528951] LustreError: 109324:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 3437.774809] LustreError: 110120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844557 with bad export cookie 12674428006465741609 [ 3437.785676] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3437.792615] LustreError: 110120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3438.248502] Lustre: server umount lustre-MDT0001 complete [ 3447.972176] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 11:29:26 (1787844566) [ 3457.121090] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3460.196782] Key type lgssc unregistered [ 3460.578023] LNet: 113775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3460.589425] LNetError: 113775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3460.612830] LNet: Removed LNI 192.168.201.106@tcp [ 3461.582840] Key type .llcrypt unregistered [ 3461.585412] Key type ._llcrypt unregistered [ 3480.583184] Key type ._llcrypt registered [ 3480.592937] Key type .llcrypt registered [ 3480.702984] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 3498.172525] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3499.462803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3499.743424] alg: No test for adler32 (adler32-zlib) [ 3500.932078] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 3501.350219] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3503.143724] Key type lgssc registered [ 3504.761970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3550.666781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3550.691801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.153732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3552.227753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3552.382233] Lustre: lustre-MDT0000: new disk, initializing [ 3552.457704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.509749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3558.877515] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3571.227738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3571.568457] Lustre: lustre-OST0000: new disk, initializing [ 3571.579551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3571.587613] Lustre: Skipped 1 previous similar message [ 3571.596280] Lustre: 117962:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3571.705127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3579.909473] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3581.623760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3581.633910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3581.658523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3592.045329] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3592.250741] Lustre: lustre-OST0001: new disk, initializing [ 3592.256645] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3592.268731] Lustre: 118826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3592.367918] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3599.421403] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3601.043559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3601.052250] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3601.078935] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3611.623243] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.633030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.641895] Lustre: Skipped 1 previous similar message [ 3615.767528] Lustre: server umount lustre-MDT0000 complete [ 3622.447940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3627.713479] LustreError: 117961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844747 with bad export cookie 14565819707735756953 [ 3627.716507] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3627.722416] LustreError: 117961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3627.785861] Lustre: server umount lustre-OST0000 complete [ 3632.080385] Lustre: server umount lustre-OST0001 complete [ 3641.836442] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 11:32:40 (1787844760) [ 3667.073225] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 11:33:05 (1787844785) [ 3677.723315] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3706.811672] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 11:33:45 (1787844825) [ 3755.533782] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 3768.756279] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3810.359060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.625361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3810.683676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3810.876692] Lustre: lustre-MDT0000: new disk, initializing [ 3811.099404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.124305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3816.623487] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3829.340242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.557281] Lustre: 126582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3829.715103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3829.737792] Lustre: Skipped 1 previous similar message [ 3829.856239] Lustre: lustre-MDT0001: new disk, initializing [ 3830.010892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3830.063364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3830.072866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3835.592366] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3846.734313] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3854.638101] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3862.573741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3863.142159] Lustre: lustre-OST0000: new disk, initializing [ 3863.147375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3863.161897] Lustre: 128553:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3863.301068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.824818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3870.854747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3871.008939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3871.567800] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3882.888273] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3889.202842] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3889.412841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3895.206591] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3895.663847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3901.925180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3901.931590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3901.942179] Lustre: Skipped 2 previous similar messages [ 3901.942528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3901.968656] Lustre: Skipped 1 previous similar message [ 3907.062694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3907.074278] Lustre: Skipped 1 previous similar message [ 3912.168750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3912.171074] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 3912.185297] Lustre: Skipped 1 previous similar message [ 3912.611644] Lustre: server umount lustre-OST0000 complete [ 3922.401408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.401791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3922.448159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3922.451374] LustreError: Skipped 1 previous similar message [ 3922.483323] Lustre: Skipped 1 previous similar message [ 3924.287960] Lustre: server umount lustre-MDT0000 complete [ 3930.424920] LustreError: 126572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845049 with bad export cookie 14565819707735758178 [ 3930.430231] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3930.433683] LustreError: 126572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3930.856675] Lustre: server umount lustre-MDT0001 complete [ 3944.232691] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 11:37:42 (1787845062) [ 3952.045151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.615550] LustreError: 130966:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.636880] LustreError: 130966:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3952.716167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3957.736428] LustreError: 130967:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3958.337988] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3961.825901] LustreError: 130966:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.945398] LustreError: 130967:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3968.624268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.797690] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3983.727733] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3991.733063] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4000.589591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4001.274278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4001.296077] Lustre: Skipped 1 previous similar message [ 4007.492898] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4016.172496] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4025.932950] Lustre: *** cfs_fail_loc=159, val=0*** [ 4025.936110] Lustre: Skipped 1 previous similar message [ 4042.542700] Lustre: lustre-MDT0000: Client 44fadb3b-dd4a-4cec-8e29-0bc6dfb41188 (at 192.168.201.6@tcp) reconnecting [ 4044.637822] Lustre: *** cfs_fail_loc=159, val=0*** [ 4044.644140] Lustre: Skipped 4 previous similar messages [ 4059.988508] Lustre: lustre-MDT0000: Client 44fadb3b-dd4a-4cec-8e29-0bc6dfb41188 (at 192.168.201.6@tcp) reconnecting [ 4065.764939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4065.778808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.795740] Lustre: Skipped 1 previous similar message [ 4065.810111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4066.738843] Lustre: server umount lustre-OST0000 complete [ 4071.392734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4071.399358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.233367] Lustre: server umount lustre-MDT0000 complete [ 4083.174286] LustreError: 130967:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.198146] LustreError: 130967:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4084.827991] LustreError: 130948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845204 with bad export cookie 14565819707735759060 [ 4084.829938] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4084.847333] LustreError: 130948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4085.319852] Lustre: server umount lustre-MDT0001 complete [ 4098.166836] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4101.311734] Key type lgssc unregistered [ 4101.670681] LNet: 134747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4101.679216] LNetError: 134747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4102.762399] LNet: Removed LNI 192.168.201.106@tcp [ 4103.869450] Key type .llcrypt unregistered [ 4103.871258] Key type ._llcrypt unregistered [ 4131.579529] Key type ._llcrypt registered [ 4131.582635] Key type .llcrypt registered [ 4131.735586] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4133.505055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4133.557548] alg: No test for adler32 (adler32-zlib) [ 4134.772290] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 4135.044592] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4136.767193] Key type lgssc registered [ 4138.230654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4148.142512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4148.191274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.135057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.401961] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4160.503474] LustreError: 136162:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4160.528631] LustreError: 136162:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4165.615828] LustreError: 136163:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.368405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.862798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4173.456578] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4183.090556] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4190.205936] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4199.510995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4199.925361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.482325] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4215.210665] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4220.481404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4230.636365] Lustre: *** cfs_fail_loc=159, val=0*** [ 4245.790031] Lustre: lustre-MDT0000: Client 399fa002-9303-4ff9-8112-6960b8c80ea9 (at 192.168.201.6@tcp) reconnecting [ 4248.553539] Lustre: *** cfs_fail_loc=159, val=0*** [ 4264.264183] Lustre: lustre-MDT0001: Client 399fa002-9303-4ff9-8112-6960b8c80ea9 (at 192.168.201.6@tcp) reconnecting [ 4266.130823] Lustre: *** cfs_fail_loc=159, val=0*** [ 4266.134161] Lustre: Skipped 3 previous similar messages [ 4282.696323] Lustre: lustre-MDT0001: Client 399fa002-9303-4ff9-8112-6960b8c80ea9 (at 192.168.201.6@tcp) reconnecting [ 4290.624046] Lustre: 139218:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4293.795900] Lustre: *** cfs_fail_loc=159, val=0*** [ 4293.797766] Lustre: Skipped 4 previous similar messages [ 4310.355502] Lustre: lustre-MDT0000: Client bfb0f400-32bf-4104-b7a1-4fa0f93904d2 (at 192.168.201.6@tcp) reconnecting [ 4312.494361] Lustre: *** cfs_fail_loc=159, val=0*** [ 4312.500441] Lustre: Skipped 14 previous similar messages [ 4328.301984] Lustre: lustre-MDT0000: Client bfb0f400-32bf-4104-b7a1-4fa0f93904d2 (at 192.168.201.6@tcp) reconnecting [ 4330.845619] Lustre: 139510:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4339.184595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4339.195839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.204894] LustreError: Skipped 1 previous similar message [ 4339.274085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4340.202459] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4340.218406] LustreError: Skipped 5 previous similar messages [ 4340.228680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4340.273531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4340.952099] Lustre: server umount lustre-OST0000 complete [ 4345.825373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4345.834738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4345.860450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4348.398189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.401076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4352.111302] Lustre: server umount lustre-MDT0000 complete [ 4353.506197] LustreError: 138934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.524613] LustreError: 138934:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4356.050919] LustreError: 139336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845475 with bad export cookie 10016426833676326783 [ 4356.057592] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4356.066608] LustreError: 139336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4356.431681] Lustre: server umount lustre-MDT0001 complete [ 4369.559188] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4373.115440] Key type lgssc unregistered [ 4373.538864] LNet: 140569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4373.543977] LNetError: 140569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4373.567205] LNet: Removed LNI 192.168.201.106@tcp [ 4374.746770] Key type .llcrypt unregistered [ 4374.753832] Key type ._llcrypt unregistered [ 4402.058201] Key type ._llcrypt registered [ 4402.062214] Key type .llcrypt registered [ 4402.172164] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4403.499536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4403.657099] alg: No test for adler32 (adler32-zlib) [ 4404.861783] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 4405.233663] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4406.927205] Key type lgssc registered [ 4408.063099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4416.712864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4416.750543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.362202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.657196] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4427.746153] LustreError: 141983:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.774658] LustreError: 141983:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4432.872459] LustreError: 141984:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.008459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.436409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4437.856937] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4446.946281] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4453.312233] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4460.746849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4461.209653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4463.340717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4468.945673] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4477.481340] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4477.720953] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4484.044979] Lustre: 144566:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4488.409704] Lustre: 144615:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4490.638891] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4491.520960] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4497.788525] Lustre: 144771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4499.074646] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4500.296708] Lustre: 144870:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4508.644438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4508.650565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4508.665635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4508.668370] Lustre: Skipped 1 previous similar message [ 4510.996804] Lustre: server umount lustre-OST0000 complete [ 4515.307149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4515.317045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.333933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4515.349041] Lustre: Skipped 1 previous similar message [ 4518.881472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.881478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4521.760631] Lustre: server umount lustre-MDT0000 complete [ 4524.013361] LustreError: 142992:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.028964] LustreError: 142992:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4526.128845] LustreError: 141963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845645 with bad export cookie 2225222626527648598 [ 4526.132850] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.144598] LustreError: 141963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4526.450885] Lustre: server umount lustre-MDT0001 complete [ 4538.126222] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4541.131360] Key type lgssc unregistered [ 4541.439965] LNet: 145930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4541.464166] LNetError: 145930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4541.500898] LNet: Removed LNI 192.168.201.106@tcp [ 4542.520835] Key type .llcrypt unregistered [ 4542.526312] Key type ._llcrypt unregistered [ 4569.404380] Key type ._llcrypt registered [ 4569.408588] Key type .llcrypt registered [ 4569.566751] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4571.162729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4571.193201] alg: No test for adler32 (adler32-zlib) [ 4572.211949] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 4572.517369] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4574.248319] Key type lgssc registered [ 4575.599094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4584.225960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4584.250228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.978284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4591.702514] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4596.200506] LustreError: 147346:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4596.239481] LustreError: 147346:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4601.316531] LustreError: 147345:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.743906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4603.551117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4609.252781] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4618.595973] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4625.924450] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4633.420266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4633.992728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4641.107609] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4648.317688] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4650.720656] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4656.509419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4661.278797] Lustre: *** cfs_fail_loc=159, val=0*** [ 4676.934794] Lustre: lustre-MDT0000: Client a7ed2729-c548-4cf3-b5e0-fba32bf666a7 (at 192.168.201.6@tcp) reconnecting [ 4684.258199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4684.266146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.296874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4687.328907] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4687.329457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4687.332569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.362798] Lustre: Skipped 1 previous similar message [ 4689.398025] Lustre: server umount lustre-OST0000 complete [ 4695.520647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4695.533241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.548868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.570313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.570868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4699.604541] Lustre: server umount lustre-MDT0000 complete [ 4702.697254] LustreError: 148078:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4702.735023] LustreError: 148078:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4703.458433] LustreError: 147327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845822 with bad export cookie 9609701341188852142 [ 4703.477317] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4703.488510] LustreError: 147327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4703.842493] Lustre: server umount lustre-MDT0001 complete [ 4715.293870] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4718.253491] Key type lgssc unregistered [ 4718.591885] LNet: 151019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4718.605051] LNetError: 151019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4718.630545] LNet: Removed LNI 192.168.201.106@tcp [ 4719.486819] Key type .llcrypt unregistered [ 4719.489544] Key type ._llcrypt unregistered [ 4746.936805] Key type ._llcrypt registered [ 4746.943322] Key type .llcrypt registered [ 4747.111898] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4748.373742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4748.428099] alg: No test for adler32 (adler32-zlib) [ 4749.704813] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 4749.952778] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4751.719152] Key type lgssc registered [ 4753.238308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4762.158743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4762.185827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.834333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.083116] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4774.375643] LustreError: 152431:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.407353] LustreError: 152431:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4778.424264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4778.912773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4784.557362] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4792.349450] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4799.056629] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4806.837863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4807.425482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.499977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4814.613654] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4822.795191] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4823.879886] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4842.924642] Lustre: lustre-OST0000: evicting 2ba7bca9-9f81-4861-9c3b-4451069215a7 (at 192.168.201.6@tcp) by administrative request [ 4842.938856] Lustre: Skipped 1 previous similar message [ 4875.232930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4875.242823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4875.249185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4875.276193] Lustre: Skipped 1 previous similar message [ 4875.293876] Lustre: Skipped 1 previous similar message [ 4878.302121] Lustre: server umount lustre-OST0000 complete [ 4885.472372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.494205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4885.507580] Lustre: Skipped 1 previous similar message [ 4888.858391] Lustre: server umount lustre-MDT0000 complete [ 4890.597291] LustreError: 152427:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.630418] LustreError: 152427:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4893.245598] LustreError: 152413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846012 with bad export cookie 5852959580261885881 [ 4893.246522] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4893.279802] LustreError: 152413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4893.883588] Lustre: server umount lustre-MDT0001 complete [ 4906.542699] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4909.922711] Key type lgssc unregistered [ 4910.336395] LNet: 156506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4910.353956] LNetError: 156506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4910.382946] LNet: Removed LNI 192.168.201.106@tcp [ 4911.675063] Key type .llcrypt unregistered [ 4911.681735] Key type ._llcrypt unregistered [ 4990.776253] Key type ._llcrypt registered [ 4990.779190] Key type .llcrypt registered [ 4990.963167] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4992.236815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4992.320067] alg: No test for adler32 (adler32-zlib) [ 4993.457510] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 4993.693360] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4995.351578] Key type lgssc registered [ 4996.684752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5005.955050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5005.976638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.335363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5011.574851] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5017.571883] LustreError: 158994:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5017.587602] LustreError: 158994:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5020.750816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.281411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5025.678709] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5034.427580] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5040.305874] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5048.700095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5049.424170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.493976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5057.397208] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5065.863188] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5067.756875] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5077.999068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5078.002199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.025489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5081.568753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5081.575869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5081.586725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5081.623304] Lustre: Skipped 1 previous similar message [ 5083.191802] Lustre: server umount lustre-OST0000 complete [ 5087.713138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5087.724060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.735085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.808098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.808502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.231244] Lustre: server umount lustre-MDT0000 complete [ 5096.928987] LustreError: 158989:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.955477] LustreError: 158989:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5098.973703] LustreError: 158976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846218 with bad export cookie 13305672763330699482 [ 5098.976103] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5098.983283] LustreError: 158976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5099.261145] Lustre: server umount lustre-MDT0001 complete [ 5112.164416] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 5115.756270] Key type lgssc unregistered [ 5116.247623] LNet: 162542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5116.257526] LNetError: 162542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5116.278211] LNet: Removed LNI 192.168.201.106@tcp [ 5117.419337] Key type .llcrypt unregistered [ 5117.425761] Key type ._llcrypt unregistered [ 5143.054537] Key type ._llcrypt registered [ 5143.059931] Key type .llcrypt registered [ 5143.182788] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5144.418480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5144.541836] alg: No test for adler32 (adler32-zlib) [ 5145.548614] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 5145.736985] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 5147.428447] Key type lgssc registered [ 5148.526404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5166.060359] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 11:58:04 (1787846284) [ 5183.355483] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5195.072395] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5238.121831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5238.168914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.549673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5239.607889] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5239.711277] Lustre: lustre-MDT0000: new disk, initializing [ 5239.866524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5239.906250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5246.640434] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5259.051510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5259.258900] Lustre: 166812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5259.357147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5259.362577] Lustre: Skipped 1 previous similar message [ 5259.508862] Lustre: lustre-MDT0001: new disk, initializing [ 5259.665419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5259.728673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5259.772701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5265.316521] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5278.890616] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5286.718100] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5296.650154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5297.093094] Lustre: lustre-OST0000: new disk, initializing [ 5297.103208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5297.118291] Lustre: 168784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5297.288728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.656400] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5306.062162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5306.076880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5306.322453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5316.385791] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5331.937254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5331.938125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.949765] LustreError: Skipped 1 previous similar message [ 5331.956284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5331.984670] Lustre: Skipped 1 previous similar message [ 5335.037524] Lustre: server umount lustre-OST0000 complete [ 5341.668089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5341.677062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5341.707044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5341.710587] Lustre: Skipped 1 previous similar message [ 5346.252871] Lustre: server umount lustre-MDT0000 complete [ 5350.654986] LustreError: 166805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846469 with bad export cookie 6671245669770129326 [ 5350.663896] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.665933] LustreError: 166805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5351.015377] Lustre: server umount lustre-MDT0001 complete [ 5365.097082] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 5368.546768] Key type lgssc unregistered [ 5368.801576] LNet: 170705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5368.812545] LNetError: 170705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5368.834963] LNet: Removed LNI 192.168.201.106@tcp [ 5369.763601] Key type .llcrypt unregistered [ 5369.767135] Key type ._llcrypt unregistered [ 5396.080576] Key type ._llcrypt registered [ 5396.082622] Key type .llcrypt registered [ 5396.167235] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5397.319844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5397.396935] alg: No test for adler32 (adler32-zlib) [ 5398.635591] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 5398.922621] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 5400.618191] Key type lgssc registered [ 5401.894677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5411.246101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5411.277213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5413.008259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5418.572805] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5423.589973] LustreError: 172120:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.599484] LustreError: 172120:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5428.129915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.496442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5433.194927] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5441.821450] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5448.279138] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5456.914234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5457.397660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.427883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5465.336442] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5476.095503] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5483.857677] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5483.874677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5483.888429] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5483.897917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.916529] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5484.982843] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5485.020271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.046878] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5485.050803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.058405] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5486.105760] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5486.120725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.135224] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5486.139541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.148829] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5487.090678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.007608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.746852] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5488.752295] Lustre: Skipped 2 previous similar messages [ 5488.768312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5488.777059] Lustre: Skipped 2 previous similar messages [ 5488.781230] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5488.788658] Lustre: Skipped 2 previous similar messages [ 5488.795824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.815480] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5488.825663] Lustre: Skipped 2 previous similar messages [ 5489.767689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.711664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.594892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.399844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.475623] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5493.486927] Lustre: Skipped 4 previous similar messages [ 5493.501531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.526459] Lustre: Skipped 4 previous similar messages [ 5493.537676] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5493.553550] Lustre: Skipped 4 previous similar messages [ 5493.561816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.567988] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5493.573190] Lustre: Skipped 4 previous similar messages [ 5494.498819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.344974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.302993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.440275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.509996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.487578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.505460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.667379] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5501.673759] Lustre: Skipped 7 previous similar messages [ 5501.686286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.697369] Lustre: Skipped 7 previous similar messages [ 5501.703721] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5501.708680] Lustre: Skipped 7 previous similar messages [ 5501.712399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.722133] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5501.730927] Lustre: Skipped 7 previous similar messages [ 5502.693653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.718823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.634224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.721236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.727036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.632993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.831844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.968334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.891298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.151137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.243583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.435271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.371501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.453737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.441319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.552992] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5518.558880] Lustre: Skipped 15 previous similar messages [ 5518.588862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5518.609637] Lustre: Skipped 15 previous similar messages [ 5518.616060] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5518.619171] Lustre: Skipped 15 previous similar messages [ 5518.634214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.645794] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5518.655153] Lustre: Skipped 15 previous similar messages [ 5519.506128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.448336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.118704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.087565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.942754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.822439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.905380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.825718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.876416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.719373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.525639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.334289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.327069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.315499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.291740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.123931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.383865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.505676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.555764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.803730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.031211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.108943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.419607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.538340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.784700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.070140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.512586] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5546.528078] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5546.535449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.811321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.931709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.162257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.153486] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5551.160445] Lustre: Skipped 30 previous similar messages [ 5551.181513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5551.194628] Lustre: Skipped 30 previous similar messages [ 5551.200589] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5551.205187] Lustre: Skipped 30 previous similar messages [ 5551.209815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.219729] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5551.229484] Lustre: Skipped 30 previous similar messages [ 5552.425121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.334763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.572724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.612683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.988713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.252691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.445644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.610035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.755190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.840142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.068139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.311424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.390850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.962620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.491330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.685356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.144643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.846669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.147647] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5575.153833] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5575.166306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.514607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.307982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.579563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.965207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.511196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.668189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.780249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.903686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.496730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.794360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.997732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.380735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.393718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.695550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.927619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.126761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.884415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.498491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.068796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.627537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.418221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.927679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.485139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.159849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.488166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.744824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.127814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.314848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.503436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.593202] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5615.599580] Lustre: Skipped 48 previous similar messages [ 5615.632170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.654714] Lustre: Skipped 48 previous similar messages [ 5615.665483] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5615.672717] Lustre: Skipped 48 previous similar messages [ 5615.678546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.692117] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5615.705221] Lustre: Skipped 48 previous similar messages [ 5616.844772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.879298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.082716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.521530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.112947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.109556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.310379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.309059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.425731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.445255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.161929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.552687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.093328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.055983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.100091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.287738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.295025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.578938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.873758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.906785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.231783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.342986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.591264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.501512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.657557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.548647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.588898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.461023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.546213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.544066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.685491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.669487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.843080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.692527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.833437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.866897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.948551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.078731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.541922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.997056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.596756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.037155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.291557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.743208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.112258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.387892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.855859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.026710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.127473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.692521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.094310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.375816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.670269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.875722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.031603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.222287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.444813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.368455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.311732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.355404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.744089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.885912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.854371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.910667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.882175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.044646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.082906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.182111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.390947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.436671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.268883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.336643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.224355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.344225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.320909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.530637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.489236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.621462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.607761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.586561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.655241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.607819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.864172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.826750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.649632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.798311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.622302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.570589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.361731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.433281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.389508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.424742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.286290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.425373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.542636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.825443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.772766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.828220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.889392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.950790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.008872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.141064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.091566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.041481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.941342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.108180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.877075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.856702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.820552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.910753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.783417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.009560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.275628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.424396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.429715] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5744.433911] Lustre: Skipped 114 previous similar messages [ 5744.460369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5744.478183] Lustre: Skipped 114 previous similar messages [ 5744.484724] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5744.491939] Lustre: Skipped 114 previous similar messages [ 5744.508549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.525855] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5744.547265] Lustre: Skipped 114 previous similar messages [ 5745.964441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.001073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.033771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.330591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.316854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.441934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.787818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.729573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.793310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.869521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.926891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.177565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.224318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.297203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.344108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.268950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.276643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.242131] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5764.261913] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5764.272596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.380273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.301886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.329531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.204723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.333610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.418884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.409737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.612380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.664948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.843204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.874914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.872228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.937903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.009409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.258703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.294391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.252287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.183325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.076544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.050385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.090462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.110977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.113269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.089608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.966838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.999128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.853801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.926380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.961942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.075586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.147524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.106883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.347370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.475080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.455543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.442464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.368162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.355308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.328878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.536501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.177727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.368471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.694145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.012358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.006702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.001111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.153750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.078163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.145602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.092678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.140388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.378805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.610083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.484874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.668457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.063540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.071951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.355444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.304928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.338254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.444738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.603789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.582817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.641523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.664284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.680153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.776029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.961388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.980274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.050751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.962751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.907701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.870843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.873519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.663415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.976972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.300751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.342603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.376624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.187962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.168478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.373949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.772384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.171082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.563384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.051125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.466973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.922356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.187135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.274878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.225860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.211177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.137769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.247726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.326577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.318486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.522705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.736731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.614918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.617640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.542835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.526341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.464286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.459542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.484467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.317655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.272075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.355506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.284126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.191676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.087579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.049776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.133043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.884469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.689664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.713110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.730124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.946918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.934500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.100415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.134928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.095337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.130973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.064705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.015259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.998712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.087201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.291713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.469895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.307530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.192443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.090914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.198240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.315369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.274230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.221713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.202486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.066630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.957540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.003030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.864162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.727499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.575319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.430217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.186470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.036054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.880287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.831052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.607316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.668586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.531895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.556154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.571572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.612207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.709297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.750626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.870194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.994513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.182658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.344677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.224832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.638377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.782578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.272306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.756951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.207673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.568269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.878047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.031324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.535866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.514081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.535611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.614759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.698662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.795222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.083432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.374855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.671460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.947038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.113922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.693197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.978180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.389799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.446403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.447051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.656651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.969521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.070037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.376474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.311595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.592682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.686641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.932880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.110962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.239893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.628614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.888573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.264548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.171774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.605920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.875239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.075319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.276399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.059835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.134577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.319168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.273795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.448889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.425730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.638238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.478497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.717165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.897951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.344447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.321700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.585991] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5998.605122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.606183] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5999.768435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.874972] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6000.886355] Lustre: Skipped 235 previous similar messages [ 6000.913380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6000.923240] Lustre: Skipped 235 previous similar messages [ 6000.927213] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6000.930995] Lustre: Skipped 235 previous similar messages [ 6000.938752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.946954] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6000.954266] Lustre: Skipped 235 previous similar messages [ 6001.789114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.002160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.234395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.207297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.053382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.996995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.837359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.702107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.780692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.762786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.800517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.613238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.495881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.431428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.477690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.912453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.113378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.542123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.626666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.600689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.649654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.810787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.783827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.788905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.770350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.911427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.882066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.910406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.892649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.049573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.911319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.000756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.979251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.848464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.706880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.541296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.371202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.345487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.263342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.030553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.898221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.811398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.804638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.687142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.667089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.588150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.587115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.549289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.434094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.356740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.190599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.055721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.144620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.028322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.136068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.916942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.966513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.839381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.816758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.676745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.435736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.456837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.325964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.367536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.260368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.265778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.488978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.555617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.580525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.488203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.236926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.305107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.224395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.375261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.240713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.193625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.972463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.970032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.838762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.665490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.418722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.522074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.672968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.729178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.081865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.104917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.048890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.118647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.087763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.006121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.014953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.173355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.058724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.009716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.978359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.008624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.845766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.936994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.943084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.853331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.653575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.733238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.670396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.807908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.072167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.948787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.967339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.733449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.784596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.667458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.472832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.379573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.239788] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6111.248026] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6111.262329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.135762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.186974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.054881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.994288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.851906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.745186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.723408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.832354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.894426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.696761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.426567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.445465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.345140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.251916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.421948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.374157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.370299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.268197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.271827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.075271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.097154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.956745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.808272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.755773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.883238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.728050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.524138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.452401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.892662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.038583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.068360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.318632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.539210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.259418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.405967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.652526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.676045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.814751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.770661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.832711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.280366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.695905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.010673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.800637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.436272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.501929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.398781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.601190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.591641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.873615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.018469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.939565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.871604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.039344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.282899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.465743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.453610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.913729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.832362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.774895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.972394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.987542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.799876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.346748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.046838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.819534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.021339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.052922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.139750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.614303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.568210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.602883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.371224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.213313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.926882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.748644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.598334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.482226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.543149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.635233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.737766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.750355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.632670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.404116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.311594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.319447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.294823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.276072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.369700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.449330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.320466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.136494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.164159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.906697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.809611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.684370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.583904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.455899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.442908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.478835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.626700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.902784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.914986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.979460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.929173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.961059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.859848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.819433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.668971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.535542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.450244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.358451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.269886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.376551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.668546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.522255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.684306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.617746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.880495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.197710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.083857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.952865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.732649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.631675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.526304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.439158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.213733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.987895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.771203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.667644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.439538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.218475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.907583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.652188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.414726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.203738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.008518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.780687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.626694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.402739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.163642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.893571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.676739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.485841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.152853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.837388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.757033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.500385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.227355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.935645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.796111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.797477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.679650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.939099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.938292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.774343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.414662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.088587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.818820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.579302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.299385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.010329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.755515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.567247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.389262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.371253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.184754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.133094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.072703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.926596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.875135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.671922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.650679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.739143] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -108 [ 6277.750187] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -108 [ 6277.770405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.777455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.754440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.715122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.734060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.513110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.347455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.223769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.204272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.914309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.837647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.640882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.498411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.332557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.110613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.908067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.026348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.300132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.532502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.653350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.742347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.070705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.987824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.988691] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6299.999488] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6300.026917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.197057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.325217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.499834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.770805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.770725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.638193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.498120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.436986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.261945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.040466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.895570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.790633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.540192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.413445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.147991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.980403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.755135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.576379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.292655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.043388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.887508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.876393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.924269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.906163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.944801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.681620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.492826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.421333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.283280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.088895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.806326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.476159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.198264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.922324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.631667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.282221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.073189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.753545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.603083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.301919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.249147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.049963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.907425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.703055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.481398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.484436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.311637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.004406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.698807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.461506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.329761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.142338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.968381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.696739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.606459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.444894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.350308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.251729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.161738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.217676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.042737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.912285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.653269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.456154] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6354.465283] LustreError: 172140:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6354.482583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.429753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.429572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.195941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.073759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.015508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.170299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.992829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.785186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.495921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.552829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.367418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.436805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.125037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.120187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.163880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.036453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.875507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.702411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.853366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.704791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.689629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.631981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.597900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.594059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.568485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.596305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.822251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.949860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.801804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.885154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.851160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.800561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.527744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.405821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.294311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.620572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.521748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.564145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.254169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.020436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.815557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.649990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.372070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.474336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.327311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.202638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.830345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.602572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.390880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.217563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.048846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.966349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.704861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.544829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.299724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.260072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.257920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.221854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.068752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.925467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.788364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.607941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.397992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.351488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.072658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.905594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.614895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.385920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.125246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.925518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.880082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.744401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.669754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.668577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.704331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.522118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.424601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.237747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.366447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.332750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.442994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.410594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.167618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.878780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.743768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.524921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.251247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.029915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.875138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.844601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.775524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.570475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.372418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.259725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.099506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.893840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.827692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.679688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.523642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.387458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.320670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.335088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.351951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.175982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.025817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.835795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.738911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.573126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.456081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.079955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.825406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.627438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.412594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.075735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.839028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.640512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.441611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.140802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.036365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.938560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.859882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.741383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.648718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.406500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.004796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.803271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.604163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.553871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.277578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.245223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.173792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.117608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.990654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.973485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.971742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.965760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.925967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.788877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.605764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.495081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.578939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.360480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.014553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.747555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.733485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.574951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.536913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.345974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.322473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.221692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.075285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.004099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.053720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.396450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.389482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.255703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.085925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.036342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.890230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.754652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6835.442798] LustreError: 181980:0:(ofd_dev.c:1783:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6835.454620] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6835.461695] LustreError: 172140:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 7310.820960] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7310.830877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.851071] Lustre: Skipped 534 previous similar messages [ 7310.870917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7312.367771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7312.375177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7314.931785] Lustre: server umount lustre-OST0000 complete [ 7323.117069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7323.134072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7323.144823] Lustre: Skipped 2 previous similar messages [ 7325.734841] Lustre: server umount lustre-MDT0000 complete [ 7328.242125] LustreError: 172120:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.263127] LustreError: 172120:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7329.946286] LustreError: 172102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848449 with bad export cookie 4228420882694232415 [ 7329.955026] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7329.965393] LustreError: 172102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7330.461243] Lustre: server umount lustre-MDT0001 complete [ 7344.302532] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 7347.405593] Key type lgssc unregistered [ 7347.721343] LNet: 227358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7347.756309] LNetError: 227358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7347.794702] LNet: Removed LNI 192.168.201.106@tcp [ 7348.914047] Key type .llcrypt unregistered [ 7348.916708] Key type ._llcrypt unregistered [ 7376.002619] Key type ._llcrypt registered [ 7376.005813] Key type .llcrypt registered [ 7376.111787] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7377.345875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7377.487255] alg: No test for adler32 (adler32-zlib) [ 7378.459465] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 7378.763897] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 7380.432864] Key type lgssc registered [ 7381.824353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7390.543519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7390.564376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.408599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7398.303591] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7401.958642] LustreError: 228779:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7401.998172] LustreError: 228779:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7407.087838] LustreError: 228778:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7407.889668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7408.530671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7415.336116] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7426.436771] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7433.984859] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7442.623229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7443.490773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7453.532556] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7461.968555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7463.599935] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7468.498995] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7470.314208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7470.333790] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7470.347074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7470.373838] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7470.377700] LustreError: 228798:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7473.121052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7473.135102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.147120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7475.505927] Lustre: lustre-OST0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 7477.230415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7477.232863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.259692] Lustre: server umount lustre-OST0000 complete [ 7490.391262] LustreError: 228798:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7514.850609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7515.246887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7517.309436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7517.353420] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7522.065742] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7527.596806] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7527.621037] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7530.593896] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7532.156233] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7538.290330] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9727498ad800.ost_server_uuid 50 [ 7540.088912] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9727498ad800.ost_server_uuid in IDLE state after 0 sec [ 7545.321667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.342973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.366189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.375627] Lustre: Skipped 1 previous similar message [ 7548.393879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7551.018653] Lustre: server umount lustre-OST0000 complete [ 7557.091792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7557.100747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.117852] Lustre: Skipped 1 previous similar message [ 7557.124225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7557.127379] Lustre: Skipped 2 previous similar messages [ 7567.846651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7567.849223] Lustre: Skipped 4 previous similar messages [ 7569.380683] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7569.835208] Lustre: server umount lustre-MDT0000 complete [ 7572.975577] LustreError: 228779:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7572.994563] LustreError: 228779:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7573.725235] LustreError: 228758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848693 with bad export cookie 4914112782383140886 [ 7573.727671] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7573.762684] LustreError: 228758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7574.501318] Lustre: server umount lustre-MDT0001 complete [ 7587.121479] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 7590.489631] Key type lgssc unregistered [ 7590.732722] LNet: 233996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7590.743496] LNetError: 233996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7590.756898] LNet: Removed LNI 192.168.201.106@tcp [ 7591.694211] Key type .llcrypt unregistered [ 7591.696053] Key type ._llcrypt unregistered [ 7628.840973] Key type ._llcrypt registered [ 7628.842318] Key type .llcrypt registered [ 7628.977880] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7630.350502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7630.391672] alg: No test for adler32 (adler32-zlib) [ 7631.606776] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 7631.932373] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 7633.591675] Key type lgssc registered [ 7634.746219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7644.039428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7644.063510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.540219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7649.808214] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7654.945622] LustreError: 235726:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7654.962841] LustreError: 235726:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7658.730190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.081198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7664.077559] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7667.715651] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7674.131543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.618267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7682.527300] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7689.966254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7691.820992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7692.234234] Lustre: lustre-OST0001: new disk, initializing [ 7692.240503] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7692.250582] Lustre: 238034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7692.444704] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7699.322690] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7701.610686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7701.617048] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7701.722151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7711.823177] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7720.582566] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7735.787736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7735.809510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.829852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7737.827179] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7737.827564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7737.854265] Lustre: Skipped 2 previous similar messages [ 7737.854704] Lustre: Skipped 2 previous similar messages [ 7740.163660] Lustre: server umount lustre-MDT0000 complete [ 7742.946281] LustreError: 237640:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7742.984917] LustreError: 237640:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7744.529844] LustreError: 235705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848863 with bad export cookie 18047488207091139917 [ 7744.538511] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7744.542313] LustreError: 235705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7745.247295] Lustre: server umount lustre-MDT0001 complete [ 7760.950916] Lustre: server umount lustre-OST0000 complete [ 7775.466450] Lustre: server umount lustre-OST0001 complete [ 7800.563838] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7812.862972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.288766] LustreError: 241143:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7813.377118] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.439202] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7818.724618] LustreError: 241144:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7822.827596] LustreError: 241143:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7827.812540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.227634] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7833.204561] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7836.477316] Lustre: 242287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7844.568081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7845.103903] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7852.126880] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7853.308459] LustreError: 242642:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7853.325806] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 7853.332209] LustreError: 242642:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7858.432215] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7861.866752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7867.413434] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 7870.124633] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7878.378298] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7882.829141] Lustre: 244157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7895.011896] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7895.019347] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7895.029921] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7895.033356] Lustre: Skipped 1 previous similar message [ 7898.090321] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7898.107732] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7898.125838] Lustre: Skipped 2 previous similar messages [ 7901.020842] Lustre: server umount mylustre-MDT0000 complete [ 7903.209960] LustreError: 241144:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.240737] LustreError: 241144:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7904.924609] LustreError: 244357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849024 with bad export cookie 18047488207091142780 [ 7904.940152] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7904.946994] LustreError: 244357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7905.464105] Lustre: server umount mylustre-MDT0001 complete [ 7920.442910] Lustre: server umount mylustre-OST0000 complete [ 7923.683095] Lustre: 234988:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849027/real 1787849027] req@ffff8ad39a5e9a40 x1874695292092160/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849043 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7923.716795] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.757296] Lustre: server umount mylustre-OST0001 complete [ 7948.942729] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7961.846605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7962.707564] LustreError: 246466:0:(ldlm_lib.c:1228:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7962.860755] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7962.868922] Lustre: Skipped 1 previous similar message [ 7968.112797] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7978.897276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7984.938645] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7988.168676] Lustre: 247607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7995.570626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.373943] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8004.148077] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 8008.255111] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8010.333617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8015.898788] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8017.562602] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8025.014719] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8028.179932] Lustre: 249478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8035.812162] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8035.818157] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8035.840572] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8035.846515] Lustre: Skipped 3 previous similar messages [ 8040.177234] Lustre: server umount tfs-MDT0000 complete [ 8041.445798] LustreError: 246466:0:(ldlm_lib.c:1228:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8041.477490] LustreError: 246466:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 8043.499920] LustreError: 246444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849162 with bad export cookie 18047488207091144957 [ 8043.501529] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8043.516703] LustreError: 246444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8043.859442] Lustre: server umount tfs-MDT0001 complete [ 8058.136934] Lustre: server umount tfs-OST0000 complete [ 8061.983275] Lustre: 234986:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787849165/real 1787849165] req@ffff8ad39dd0de00 x1874695292142976/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787849181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8062.027371] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8062.044441] Lustre: Skipped 3 previous similar messages [ 8062.325605] Lustre: server umount tfs-OST0001 complete [ 8083.662698] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8093.561409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8094.297993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.310422] Lustre: Skipped 3 previous similar messages [ 8100.121239] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8109.530367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8115.263647] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8118.356951] Lustre: 252871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8126.098573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8133.294781] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8134.982710] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 8139.145126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8142.299656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8147.977298] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8150.868492] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8159.105712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8163.020824] Lustre: 254743:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8169.606267] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 12:48:08 (1787849288) [ 8186.848678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8186.853258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.873170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8186.890227] Lustre: Skipped 4 previous similar messages [ 8201.183167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8201.860500] Lustre: server umount lustre-MDT0000 complete [ 8204.264531] LustreError: 251728:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8204.295236] LustreError: 251728:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 8206.118853] LustreError: 251706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849325 with bad export cookie 18047488207091146875 [ 8206.125259] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8206.136235] LustreError: 251706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8209.379397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8209.386729] Lustre: Skipped 13 previous similar messages [ 8212.955967] Lustre: server umount lustre-MDT0001 complete [ 8224.716905] Lustre: server umount lustre-OST0000 complete [ 8236.381611] Lustre: server umount lustre-OST0001 complete [ 8243.391881] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8254.230515] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8292.337522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8292.695582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8292.744849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8292.807430] Lustre: lustre-MDT0000: new disk, initializing [ 8292.860716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8292.865019] Lustre: Skipped 3 previous similar messages [ 8292.875438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8297.603199] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8309.897373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.091089] Lustre: 257932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8310.218358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8310.221022] Lustre: Skipped 1 previous similar message [ 8310.377273] Lustre: lustre-MDT0001: new disk, initializing [ 8310.577861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8310.588600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8315.742109] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8327.444059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8327.890639] Lustre: lustre-OST0000: new disk, initializing [ 8327.902551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8327.919121] Lustre: 259510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8329.242812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8329.260795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8329.402829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8335.580423] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8347.197416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8347.385434] Lustre: lustre-OST0001: new disk, initializing [ 8347.388134] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8347.393053] Lustre: 260383:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8349.317750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8349.345334] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8349.577191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8354.834622] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8367.964405] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 12:51:25 (1787849485) [ 8389.187728] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 12:51:46 (1787849506) [ 8411.116226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8411.117487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.145918] Lustre: Skipped 2 previous similar messages [ 8411.167712] Lustre: Skipped 6 previous similar messages [ 8423.393753] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8423.516994] Lustre: server umount lustre-OST0000 complete [ 8428.512391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8434.979437] Lustre: server umount lustre-MDT0000 complete [ 8439.679077] LustreError: 257922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849559 with bad export cookie 18047488207091148653 [ 8439.693307] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8439.694923] LustreError: 257922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8454.111153] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8454.688250] Lustre: server umount lustre-MDT0001 complete [ 8466.400241] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8538.069407] Lustre: server umount lustre-OST0001 complete [ 8546.667028] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8564.290685] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8603.575588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8603.880830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8603.913528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8603.993888] Lustre: lustre-MDT0000: new disk, initializing [ 8604.067255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8604.074606] Lustre: Skipped 3 previous similar messages [ 8604.096763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8609.353707] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8621.154446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8621.320543] Lustre: 265233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8621.388030] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8621.392178] Lustre: Skipped 1 previous similar message [ 8621.464751] Lustre: lustre-MDT0001: new disk, initializing [ 8621.607385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8621.617627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8626.563278] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8637.518873] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8642.953602] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8649.667469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8649.957413] Lustre: 267206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8651.138958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8651.221803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8655.527621] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8665.652293] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8676.837141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.840522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8676.841647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8676.851665] Lustre: Skipped 5 previous similar messages [ 8676.868400] Lustre: Skipped 13 previous similar messages [ 8678.308623] Lustre: server umount lustre-OST0000 complete [ 8688.379991] Lustre: server umount lustre-MDT0000 complete [ 8692.060529] LustreError: 265226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849811 with bad export cookie 18047488207091150123 [ 8692.069147] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8692.070390] LustreError: 265226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8692.433033] Lustre: server umount lustre-MDT0001 complete [ 8703.788234] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8706.334613] Key type lgssc unregistered [ 8706.605814] LNet: 268968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8706.615170] LNetError: 268968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8706.626414] LNet: Removed LNI 192.168.201.106@tcp [ 8707.650308] Key type .llcrypt unregistered [ 8707.651819] Key type ._llcrypt unregistered [ 8734.172642] Key type ._llcrypt registered [ 8734.175656] Key type .llcrypt registered [ 8734.312531] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8735.389865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8735.583075] alg: No test for adler32 (adler32-zlib) [ 8736.792764] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 8737.199182] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8739.071208] Key type lgssc registered [ 8740.739652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8754.001989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8754.051874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8755.933488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.096983] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8766.444280] LustreError: 270432:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.480536] LustreError: 270432:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8771.563904] LustreError: 270433:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.688547] LustreError: 270432:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8779.624978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.168279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8785.393568] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8793.015017] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8798.830471] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8807.950937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8808.487877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8815.681242] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8823.934297] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8825.001842] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8825.908507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8830.947978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8830.952042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8830.952750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8830.968040] LustreError: Skipped 1 previous similar message [ 8831.016609] Lustre: Skipped 1 previous similar message [ 8833.429414] Lustre: server umount lustre-OST0000 complete [ 8841.187616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.204772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8841.219963] Lustre: Skipped 2 previous similar messages [ 8843.331237] Lustre: server umount lustre-MDT0000 complete [ 8846.306234] LustreError: 270432:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.330613] LustreError: 270432:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8847.067376] LustreError: 270412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849966 with bad export cookie 624794699698386461 [ 8847.085835] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8847.341446] Lustre: server umount lustre-MDT0001 complete [ 8859.098966] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8862.187664] Key type lgssc unregistered [ 8862.551078] LNet: 274024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8862.560530] LNetError: 274024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8862.590541] LNet: Removed LNI 192.168.201.106@tcp [ 8863.602131] Key type .llcrypt unregistered [ 8863.603874] Key type ._llcrypt unregistered [ 8881.332013] Key type ._llcrypt registered [ 8881.334731] Key type .llcrypt registered [ 8881.430933] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8882.469899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8882.648933] alg: No test for adler32 (adler32-zlib) [ 8883.701109] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 8884.010292] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8885.703284] Key type lgssc registered [ 8886.595523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8897.049987] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8897.066735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.607477] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8917.727332] LustreError: 275219:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.106@tcp: failed processing log, type 4: rc = -110 [ 8944.415340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8951.434643] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8960.861131] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8963.169640] Lustre: server umount lustre-OST0000 complete [ 8981.842567] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8984.948507] Key type lgssc unregistered [ 8985.249936] LNet: 276869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8985.255632] LNetError: 276869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8985.274794] LNet: Removed LNI 192.168.201.106@tcp [ 8986.203492] Key type .llcrypt unregistered [ 8986.209746] Key type ._llcrypt unregistered [ 9027.052368] Key type ._llcrypt registered [ 9027.054711] Key type .llcrypt registered [ 9027.159515] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9028.524932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9028.571651] alg: No test for adler32 (adler32-zlib) [ 9029.618432] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 9029.890222] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9031.591232] Key type lgssc registered [ 9032.829816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9046.775071] loop0: detected capacity change from 0 to 204800000 [ 9046.979354] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9046.989510] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9047.352599] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9048.551733] loop0: detected capacity change from 0 to 204800000 [ 9048.748964] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9048.758913] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9049.084551] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9050.188954] loop0: detected capacity change from 0 to 204800000 [ 9050.336167] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9050.343886] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9050.456293] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9051.736103] loop0: detected capacity change from 0 to 204800000 [ 9051.917610] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9052.036696] print_req_error: 1 callbacks suppressed [ 9052.036710] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9053.078215] loop0: detected capacity change from 0 to 204800000 [ 9053.115022] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9058.137386] loop0: detected capacity change from 0 to 204800000 [ 9058.176599] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9062.067195] loop0: detected capacity change from 0 to 204800000 [ 9062.090433] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9066.619438] loop0: detected capacity change from 0 to 204800000 [ 9066.657815] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9070.860751] loop0: detected capacity change from 0 to 204800000 [ 9070.983708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9071.087095] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9071.259473] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9071.283992] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9071.307155] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9071.325050] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9071.370716] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9071.434240] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9071.477640] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9071.492248] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9071.506800] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9071.530205] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9071.542895] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9071.559397] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9071.588291] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9071.609922] Lustre: 280284:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9078.417077] Lustre: server umount MGS complete [ 9079.472500] loop0: detected capacity change from 0 to 204800000 [ 9079.536557] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9080.112274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.172989] Lustre: lustre-MDT0000: Aborting client recovery [ 9081.349798] loop1: detected capacity change from 0 to 204800000 [ 9081.595851] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.721498] Lustre: 280716:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9081.748086] Lustre: 280716:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9081.773875] Lustre: 280716:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9081.792470] Lustre: 280716:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9081.816189] Lustre: 280716:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9081.834278] Lustre: 280716:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9081.852122] Lustre: 280716:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9081.885161] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9082.204977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9082.259797] Lustre: lustre-MDT0001: Aborting client recovery [ 9083.537170] loop2: detected capacity change from 0 to 204800000 [ 9083.851820] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9083.922623] Lustre: 280795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9083.940829] Lustre: 280795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9083.970844] Lustre: 280795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9083.995911] Lustre: 280795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9084.007153] Lustre: 280795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9084.029808] Lustre: 280795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9084.044407] Lustre: 280795:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9084.254919] Lustre: Found index 0 for lustre-OST0000, updating log [ 9084.285448] Lustre: 280814:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9084.390881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.415504] Lustre: lustre-OST0000: Aborting recovery [ 9085.534350] loop3: detected capacity change from 0 to 204800000 [ 9085.651488] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9085.682318] Lustre: 280879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9085.689076] Lustre: 280879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9085.699026] Lustre: 280879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9085.707095] Lustre: 280879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9085.712114] Lustre: 280879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9085.717985] Lustre: 280879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9085.724912] Lustre: 280879:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9085.782509] Lustre: Found index 1 for lustre-OST0001, updating log [ 9085.875623] Lustre: 280885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9085.951958] Lustre: lustre-OST0001: Aborting recovery [ 9090.020510] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9090.027428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9090.051829] LustreError: 280659:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9090.077422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9090.112400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9092.066172] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9092.067100] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9093.993467] LustreError: 280626:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 603us: rc = -2 [ 9094.543944] LustreError: 280626:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 430us: rc = -2 [ 9094.555525] LustreError: 280626:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 9107.446064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.470283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9107.478280] Lustre: Skipped 3 previous similar messages [ 9112.545758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9112.561510] Lustre: Skipped 3 previous similar messages [ 9113.858632] Lustre: server umount lustre-MDT0000 complete [ 9114.760923] LustreError: 280608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850234 with bad export cookie 14790633848086410241 [ 9114.783179] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9117.667336] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9117.669412] LustreError: 280625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9117.672157] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9117.687026] Lustre: Skipped 4 previous similar messages [ 9117.733318] LustreError: 280625:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9119.712730] LustreError: 281164:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9119.713284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9119.748970] LustreError: 281164:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9119.759498] Lustre: Skipped 2 previous similar messages [ 9121.315255] Lustre: server umount lustre-MDT0001 complete [ 9128.771797] Lustre: server umount lustre-OST0000 complete [ 9144.288273] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9144.651483] Lustre: server umount lustre-OST0001 complete [ 9153.749204] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 13:04:32 (1787850272) [ 9185.223432] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9197.187677] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9238.349720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9238.691988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9238.737652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9238.836963] Lustre: lustre-MDT0000: new disk, initializing [ 9238.922320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9238.925493] Lustre: Skipped 1 previous similar message [ 9238.946324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9243.969529] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9255.100873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.355862] Lustre: 284959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9255.415507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9255.419076] Lustre: Skipped 1 previous similar message [ 9255.472377] Lustre: lustre-MDT0001: new disk, initializing [ 9255.575706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9255.629542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9255.645783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9260.816279] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9271.532334] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9277.801462] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9285.104754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9285.385401] Lustre: lustre-OST0000: new disk, initializing [ 9285.392206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9285.404188] Lustre: 286927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9285.489494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9292.546867] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9293.927476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9293.951584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9294.151384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9303.147256] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9308.592233] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9320.664501] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9324.933889] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9332.390589] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9365.984623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9365.988094] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.992410] LustreError: Skipped 1 previous similar message [ 9365.999120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9366.029131] Lustre: Skipped 1 previous similar message [ 9369.320771] Lustre: server umount lustre-OST0000 complete [ 9373.670915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9373.678452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9376.230315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9376.247330] Lustre: Skipped 3 previous similar messages [ 9379.320761] Lustre: server umount lustre-MDT0000 complete [ 9381.353210] LustreError: 285825:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9381.373900] LustreError: 285825:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9384.112942] LustreError: 288538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850503 with bad export cookie 14790633848086412957 [ 9384.115990] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9384.133779] LustreError: 288538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9384.785210] Lustre: server umount lustre-MDT0001 complete [ 9393.240267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9397.999198] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9403.609282] Lustre: server umount MGS complete [ 9416.734936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.161474] LustreError: 290738:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9417.172286] LustreError: 290751:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9417.241750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.224688] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9422.318123] LustreError: 290739:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.557733] LustreError: 290739:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9432.581095] LustreError: 290739:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9432.718179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9439.056950] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9448.787752] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9455.769753] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9464.118644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9464.441303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9464.449463] Lustre: Skipped 1 previous similar message [ 9472.453440] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9481.699099] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9518.051779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9518.058377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9518.075853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9518.101979] Lustre: Skipped 2 previous similar messages [ 9518.122337] Lustre: Skipped 1 previous similar message [ 9529.311312] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9529.570088] Lustre: server umount lustre-OST0000 complete [ 9535.456306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9535.466872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9541.016307] Lustre: server umount lustre-MDT0000 complete [ 9543.651490] LustreError: 290739:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9543.676781] LustreError: 290739:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9545.931076] LustreError: 290719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850665 with bad export cookie 14790633848086414735 [ 9545.940886] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9545.947717] LustreError: 290719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9546.346087] Lustre: server umount lustre-MDT0001 complete [ 9559.401286] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9562.561792] Key type lgssc unregistered [ 9562.903837] LNet: 294692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9562.909378] LNetError: 294692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9562.923415] LNet: Removed LNI 192.168.201.106@tcp [ 9563.764810] Key type .llcrypt unregistered [ 9563.766889] Key type ._llcrypt unregistered [ 9605.212074] Key type ._llcrypt registered [ 9605.213519] Key type .llcrypt registered [ 9605.371178] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9624.419163] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9625.732941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9625.761986] alg: No test for adler32 (adler32-zlib) [ 9626.899215] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 9627.241958] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9628.975382] Key type lgssc registered [ 9630.347422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9673.512801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9673.559928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9674.890634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9674.931176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9674.998367] Lustre: lustre-MDT0000: new disk, initializing [ 9675.190329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9675.217495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9679.210275] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9691.824438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9692.115578] Lustre: 298656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9692.220085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9692.222631] Lustre: Skipped 1 previous similar message [ 9692.284270] Lustre: lustre-MDT0001: new disk, initializing [ 9692.394569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9692.448903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9692.466093] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9697.643955] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9710.431705] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9720.522901] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9730.459996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9730.704569] Lustre: lustre-OST0000: new disk, initializing [ 9730.708507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9730.715592] Lustre: 300628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9730.865371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9738.579463] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9739.942890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9739.964465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9740.267994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9751.759139] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9759.376630] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9773.299486] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9781.263447] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9837.538674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9837.556717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9837.558753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9837.599643] LustreError: Skipped 1 previous similar message [ 9842.684896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9842.694153] Lustre: Skipped 2 previous similar messages [ 9847.777922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9847.781083] Lustre: Skipped 1 previous similar message [ 9851.359153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9851.720688] Lustre: server umount lustre-OST0000 complete [ 9856.498911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9856.510151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9856.544660] Lustre: Skipped 1 previous similar message [ 9856.551522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9858.020890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9862.354695] Lustre: server umount lustre-MDT0000 complete [ 9866.772305] LustreError: 298648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850986 with bad export cookie 3840454558966128970 [ 9866.779540] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9866.787326] LustreError: 298648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9867.584377] Lustre: server umount lustre-MDT0001 complete [ 9878.158889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.948825] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9889.673378] Lustre: server umount MGS complete [ 9901.518285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.028440] LustreError: 304518:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9902.045417] LustreError: 304518:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9902.056574] LustreError: 304531:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9902.104995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9906.564594] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9907.176449] LustreError: 304519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.292303] LustreError: 304518:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9916.437432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9916.973501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9922.579161] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9931.810278] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9938.730487] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9948.775131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9949.206325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9956.058703] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9964.591602] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10007.521927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10007.529990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10007.542554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10007.564591] Lustre: Skipped 1 previous similar message [10007.596093] Lustre: Skipped 3 previous similar messages [10012.551516] Lustre: server umount lustre-OST0000 complete [10019.298589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10019.311428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.329523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10025.414936] Lustre: server umount lustre-MDT0000 complete [10027.073977] LustreError: 304514:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10027.103279] LustreError: 304514:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10030.807334] LustreError: 304498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851150 with bad export cookie 3840454558966130811 [10030.817612] LustreError: 304498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10030.834136] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10031.245818] Lustre: server umount lustre-MDT0001 complete [10044.989328] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [10048.004827] Key type lgssc unregistered [10048.299283] LNet: 308493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10048.310749] LNetError: 308493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10049.386686] LNet: Removed LNI 192.168.201.106@tcp [10050.631614] Key type .llcrypt unregistered [10050.633317] Key type ._llcrypt unregistered [10074.365295] Key type ._llcrypt registered [10074.370530] Key type .llcrypt registered [10074.575051] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [10089.592140] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10090.993449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10091.273143] alg: No test for adler32 (adler32-zlib) [10092.430526] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [10092.844612] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [10094.583246] Key type lgssc registered [10096.242841] Lustre: Echo OBD driver; http://www.lustre.org/ [10125.851870] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10142.589274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10142.619751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.957811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10143.987463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10144.137651] Lustre: lustre-MDT0000: new disk, initializing [10144.310633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10144.351409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10149.548793] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10162.218944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10162.669980] Lustre: lustre-OST0000: new disk, initializing [10162.676107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10162.683582] Lustre: Skipped 1 previous similar message [10162.692974] Lustre: 312545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10162.804560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10169.076052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10169.090846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10169.138446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10172.270852] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11044.833031] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11044.855853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11049.454180] Lustre: server umount lustre-MDT0000 complete [11053.516450] LustreError: 312544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852172 with bad export cookie 4792182569888614242 [11053.517239] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11053.529974] LustreError: 312544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11053.741411] Lustre: server umount lustre-OST0000 complete [11086.885281] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [11100.909453] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11149.990895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11150.284407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11150.314385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11150.410560] Lustre: lustre-MDT0000: new disk, initializing [11150.501516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11150.515439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11155.313923] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11166.149381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.242214] Lustre: 316887:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11166.284456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11166.287272] Lustre: Skipped 1 previous similar message [11166.334611] Lustre: lustre-MDT0001: new disk, initializing [11166.427333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11166.493984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11166.519249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11171.185799] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11182.255304] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11189.075588] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11199.128124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11199.573444] Lustre: lustre-OST0000: new disk, initializing [11199.578235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11199.585507] Lustre: 318856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11199.726023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11207.818764] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11209.778535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11209.800026] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11209.921249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11218.364712] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11223.988513] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11224.400766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11229.905212] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11230.266797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11235.812788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11235.823584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11235.846284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11235.899838] Lustre: Skipped 1 previous similar message [11240.933911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11240.942465] Lustre: Skipped 2 previous similar messages [11246.071792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11246.089619] Lustre: Skipped 1 previous similar message [11247.073847] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11247.473686] Lustre: server umount lustre-OST0000 complete [11256.288874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11256.300890] Lustre: Skipped 1 previous similar message [11256.301640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11256.312256] Lustre: Skipped 1 previous similar message [11262.200423] Lustre: server umount lustre-MDT0000 complete [11266.805301] LustreError: 316878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852386 with bad export cookie 4792182569891555068 [11266.808344] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11266.815860] LustreError: 316878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11267.389319] Lustre: server umount lustre-MDT0001 complete [11271.498497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11274.169443] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 13:39:52 (1787852392) [11281.045940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11281.452917] LustreError: 321175:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11281.478862] LustreError: 321175:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11281.536677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11286.002301] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11287.024059] LustreError: 321176:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11291.109694] LustreError: 321175:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11294.266263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11294.769531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11299.856051] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11308.232676] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11313.622582] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11320.116258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11320.412517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11326.348557] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11334.946417] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11342.522346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11342.768381] Lustre: lustre-OST0001: new disk, initializing [11342.776166] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11342.791990] Lustre: 324010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11342.933047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11349.986827] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11352.221544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11352.256946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11352.310429] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11353.391164] LustreError: 324471:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11353.412603] LustreError: 324471:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11354.411755] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11356.520796] LustreError: 324471:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11356.538407] LustreError: 324471:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [11360.818129] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11360.826655] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [11362.757797] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11363.109488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11369.072709] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11369.080632] LustreError: 324476:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11370.581469] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff97274a62e000.ost_server_uuid 50 [11372.258657] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff97274a62e000.ost_server_uuid in FULL state after 0 sec [11378.246211] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97274a62e000.ost_server_uuid 50 [11379.983324] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97274a62e000.ost_server_uuid in IDLE state after 0 sec [11381.444599] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11396.890685] Lustre: lustre-OST0001: Not available for connect from 192.168.201.6@tcp (stopping) [11396.910428] Lustre: Skipped 2 previous similar messages [11397.095584] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11397.096668] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11397.105409] LustreError: Skipped 1 previous similar message [11397.140101] Lustre: Skipped 1 previous similar message [11408.863716] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11408.950055] Lustre: server umount lustre-OST0001 complete [11412.215546] LustreError: 322996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11416.032204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11416.039483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11416.060568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11416.065755] Lustre: Skipped 10 previous similar messages [11419.106900] LustreError: 322993:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11419.159132] LustreError: 322993:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11422.491480] Lustre: server umount lustre-OST0000 complete [11429.361373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11429.416413] Lustre: Skipped 2 previous similar messages [11435.201433] Lustre: server umount lustre-MDT0000 complete [11439.585676] LustreError: 321170:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11439.614577] LustreError: 321170:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11441.680873] LustreError: 321156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852561 with bad export cookie 4792182569891555943 [11441.700713] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11441.703329] LustreError: 321156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11442.224941] Lustre: server umount lustre-MDT0001 complete [11457.253680] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11460.456986] Key type lgssc unregistered [11460.736288] LNet: 326709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11460.744785] LNetError: 326709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11460.762484] LNet: Removed LNI 192.168.201.106@tcp [11461.559601] Key type .llcrypt unregistered [11461.564353] Key type ._llcrypt unregistered [11493.153724] Key type ._llcrypt registered [11493.155964] Key type .llcrypt registered [11493.360415] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11495.516943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11495.666610] alg: No test for adler32 (adler32-zlib) [11497.092270] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [11497.442909] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11499.127173] Key type lgssc registered [11501.683663] Lustre: Echo OBD driver; http://www.lustre.org/ [11511.250348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11511.276470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11512.898319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11518.101755] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11523.046123] LustreError: 328130:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.066791] LustreError: 328130:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11525.294649] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11528.189406] LustreError: 328131:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11531.512830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11531.851124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11535.803624] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11546.547728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11547.229536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11557.066474] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11557.418386] LustreError: 329811:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.477511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11563.492849] LustreError: 329811:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11563.534673] LustreError: 329811:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11568.233889] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11568.618201] LustreError: 329813:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11568.637323] LustreError: 329813:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11576.646362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11577.093216] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11582.467528] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11585.910492] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11596.271996] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11605.100810] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11605.565240] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11710.444758] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11710.445583] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.477205] LustreError: Skipped 1 previous similar message [11710.510420] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11715.563905] LustreError: 329812:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11715.579169] LustreError: 329812:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11715.657522] Lustre: server umount lustre-OST0001 complete [11725.795177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11725.837387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11725.847346] Lustre: Skipped 2 previous similar messages [11725.894970] Lustre: Skipped 2 previous similar messages [11730.766399] Lustre: server umount lustre-OST0000 complete [11736.011344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11736.013695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11736.028330] Lustre: Skipped 1 previous similar message [11736.040638] Lustre: Skipped 1 previous similar message [11741.152230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11741.161745] Lustre: Skipped 1 previous similar message [11746.273729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11750.367175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11750.695617] Lustre: server umount lustre-MDT0000 complete [11751.393033] LustreError: 329053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11751.412562] LustreError: 329053:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11754.268624] LustreError: 328110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852873 with bad export cookie 9488068406165327276 [11754.271330] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11754.275564] LustreError: 328110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11754.647698] Lustre: server umount lustre-MDT0001 complete [11767.462528] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11771.052846] Key type lgssc unregistered [11771.343371] LNet: 333079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11771.349813] LNetError: 333079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11771.381696] LNet: Removed LNI 192.168.201.106@tcp [11772.760430] Key type .llcrypt unregistered [11772.767952] Key type ._llcrypt unregistered [11828.882385] Key type ._llcrypt registered [11828.886284] Key type .llcrypt registered [11828.998904] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11830.829190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11830.929755] alg: No test for adler32 (adler32-zlib) [11832.187222] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [11832.414859] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11834.095260] Key type lgssc registered [11835.303990] Lustre: Echo OBD driver; http://www.lustre.org/ [11846.455580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11846.474760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11848.071502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11853.213114] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11858.405766] LustreError: 335373:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.429199] LustreError: 335373:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11861.913646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11862.279062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11867.026734] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11870.143739] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11876.818203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11877.312553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11883.775955] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11887.531907] LustreError: 336864:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11892.732950] LustreError: 336865:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11893.835522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11893.886518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11894.321305] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11899.393633] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11901.704788] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11911.243467] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11921.848050] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11935.206941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11935.218774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11935.234576] Lustre: Skipped 3 previous similar messages [11935.236809] Lustre: Skipped 2 previous similar messages [11940.328921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11940.335033] Lustre: Skipped 4 previous similar messages [11940.684484] Lustre: server umount lustre-MDT0000 complete [11944.531803] LustreError: 335352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853063 with bad export cookie 6596649236635900981 [11944.537174] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11944.544761] LustreError: 335352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11944.786854] Lustre: server umount lustre-MDT0001 complete [11958.903587] Lustre: server umount lustre-OST0000 complete [11960.672329] Lustre: 334637:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853064/real 1787853064] req@ffff8ad392b31e00 x1874699696014080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787853080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11960.703556] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11964.254264] Lustre: server umount lustre-OST0001 complete [11984.640646] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11996.408564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11996.746299] LustreError: 340638:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11996.765898] LustreError: 340638:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11996.838246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12001.286670] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12002.279261] LustreError: 340639:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12010.174256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12010.589804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12015.430915] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12018.818055] Lustre: 341779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12027.030568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12027.700520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12034.871575] LustreError: 342132:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12034.912460] LustreError: 342132:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12036.199544] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12042.291496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12047.248766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12053.023271] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12055.907751] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12065.342426] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12070.994209] Lustre: 343652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12077.027523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12077.038147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12077.056518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12082.567507] Lustre: server umount lustre-MDT0000 complete [12083.685995] LustreError: 340639:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.708202] LustreError: 340639:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [12087.331701] LustreError: 340620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853206 with bad export cookie 6596649236635902682 [12087.334799] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12087.343190] LustreError: 340620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12087.848949] Lustre: server umount lustre-MDT0001 complete [12102.966028] Lustre: server umount lustre-OST0000 complete [12105.191533] Lustre: 334635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787853208/real 1787853208] req@ffff8ad39571c3c0 x1874699696068736/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787853224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12105.233802] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12105.251691] Lustre: Skipped 3 previous similar messages [12107.595497] Lustre: server umount lustre-OST0001 complete [12112.106497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12114.142954] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12116.064901] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 13:53:54 (1787853234) [12145.362911] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12156.342435] loop0: detected capacity change from 0 to 1099511627776 [12156.702216] SGI XFS with ACLs, security attributes, quota, no debug enabled [12156.707763] *** VALIDATE xfs *** [12156.713236] XFS (loop0): Mounting V5 Filesystem [12156.732693] XFS (loop0): Ending clean mount [12158.494675] loop1: detected capacity change from 0 to 18691697672192 [12295.842529] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [12306.352578] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12345.156146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12345.448165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12345.469552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12345.544931] Lustre: lustre-MDT0000: new disk, initializing [12345.628720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12345.631713] Lustre: Skipped 1 previous similar message [12345.655737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12350.943377] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12362.499330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12362.653576] Lustre: 348794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12362.725497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12362.733296] Lustre: Skipped 1 previous similar message [12362.807859] Lustre: lustre-MDT0001: new disk, initializing [12363.014983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12363.034427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12368.605541] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12379.278559] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12386.380955] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12394.028082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12394.465182] Lustre: lustre-OST0000: new disk, initializing [12394.476826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12394.487854] Lustre: 350764:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12395.970399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12395.988728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12396.106972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12401.598248] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12412.248436] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12417.933566] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12418.158627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12423.855447] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12424.216486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12426.723085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12426.726840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12426.747329] Lustre: Skipped 1 previous similar message [12426.766786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12426.773373] Lustre: Skipped 5 previous similar messages [12431.841420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12431.864150] Lustre: Skipped 1 previous similar message [12432.673555] Lustre: server umount lustre-OST0000 complete [12436.963078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12436.983671] Lustre: Skipped 1 previous similar message [12442.085977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12442.090196] Lustre: Skipped 3 previous similar messages [12451.295584] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12451.873522] Lustre: server umount lustre-MDT0000 complete [12452.329157] LustreError: 348807:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12452.360556] LustreError: 348807:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12457.588907] LustreError: 348787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853576 with bad export cookie 6596649236635904236 [12457.598682] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12457.606933] LustreError: 348787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12458.166373] Lustre: server umount lustre-MDT0001 complete [12467.992766] XFS (loop0): Unmounting Filesystem [12473.740405] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 13:59:52 (1787853592) [12481.709254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.200388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12482.207647] Lustre: Skipped 2 previous similar messages [12487.202868] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12495.834737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12500.761657] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12508.702967] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12514.367811] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12521.813120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12528.620669] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12536.454675] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12552.160565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12552.169143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12552.174888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12552.186638] Lustre: Skipped 1 previous similar message [12552.204755] Lustre: Skipped 2 previous similar messages [12554.816561] Lustre: server umount lustre-OST0000 complete [12565.907143] Lustre: server umount lustre-MDT0000 complete [12567.532601] LustreError: 353330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12567.567466] LustreError: 353330:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [12569.743591] LustreError: 354064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853689 with bad export cookie 6596649236635905118 [12569.753530] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12569.765908] LustreError: 354064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12570.280385] Lustre: server umount lustre-MDT0001 complete [12582.918411] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12586.278523] Key type lgssc unregistered [12586.702094] LNet: 356974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12586.711266] LNetError: 356974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12586.738135] LNet: Removed LNI 192.168.201.106@tcp [12587.694176] Key type .llcrypt unregistered [12587.699847] Key type ._llcrypt unregistered [12617.280352] Key type ._llcrypt registered [12617.282550] Key type .llcrypt registered [12617.371176] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12618.349689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12618.362685] alg: No test for adler32 (adler32-zlib) [12619.681058] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [12620.041104] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12621.751251] Key type lgssc registered [12623.038428] Lustre: Echo OBD driver; http://www.lustre.org/ [12631.665544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12631.689146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12633.260192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12639.094687] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12642.800402] LustreError: 358542:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12642.825390] LustreError: 358542:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12647.907219] LustreError: 358543:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12649.036581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12649.484867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12654.171864] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12662.807231] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12668.482751] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12676.605067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12677.364838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12683.932810] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12693.455369] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12705.249120] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12705.288628] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12705.303570] Lustre: Skipped 1 previous similar message [12707.095443] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [12710.373218] LustreError: 359270:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12710.416242] LustreError: 359270:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12710.481754] Lustre: server umount lustre-MDT0001 complete [12715.494110] LustreError: 358538:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12715.540936] LustreError: 358538:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12720.613985] LustreError: 358537:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12720.645498] LustreError: 358537:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12722.984555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12723.183793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12723.190100] Lustre: Found index 1 for lustre-MDT0001, updating log [12723.265179] Lustre: 358530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12723.718985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12728.852321] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12729.549105] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12732.416461] LustreError: 362115:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12732.421598] LustreError: 362115:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12732.429815] LustreError: 358534:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12732.534127] LustreError: 362124:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.106@tcp: cfg command failed: rc = -17 [12732.552268] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12732.552268] [12742.941757] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [12744.176698] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12744.185415] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12744.221473] Lustre: Skipped 1 previous similar message [12747.396538] Lustre: server umount lustre-MDT0001 complete [12748.023390] LustreError: 358538:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12748.071865] LustreError: 358538:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12759.213428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12759.365821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12759.370341] Lustre: Found index 1 for lustre-MDT0001, updating log [12759.421618] Lustre: 358531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12759.751043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12765.180761] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12765.341526] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12767.802162] LustreError: 363176:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12767.823436] LustreError: 363176:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12767.852907] LustreError: 358534:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12767.871118] LustreError: 358534:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12767.987830] LustreError: 363188:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12767.996289] LustreError: 363188:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12768.001672] LustreError: 363188:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.106@tcp: cfg command failed: rc = -17 [12768.009530] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.106@tcp [12768.009530] [13075.725192] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [13075.732319] Lustre: Skipped 2 previous similar messages [13077.477177] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13077.503471] Lustre: Skipped 1 previous similar message [13077.512101] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13080.721686] Lustre: server umount lustre-MDT0001 complete [13080.836075] LustreError: 360959:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13080.852572] LustreError: 360959:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [13091.490769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13091.561710] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13091.571251] Lustre: Found index 1 for lustre-MDT0001, updating log [13091.624603] Lustre: 358531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13091.923667] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13096.595171] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13096.946478] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13100.018453] LustreError: 371593:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13100.024161] LustreError: 371593:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13100.030494] LustreError: 358534:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13100.034192] LustreError: 358534:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13100.088400] LustreError: 371596:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13100.104452] LustreError: 371596:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.106@tcp: cfg command failed: rc = -114 [13100.123469] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13100.123469] [13419.492566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13419.498613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13419.519240] Lustre: Skipped 1 previous similar message [13419.532134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13419.532142] Lustre: Skipped 2 previous similar messages [13423.774991] Lustre: server umount lustre-MDT0000 complete [13424.616825] LustreError: 369433:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13424.641075] LustreError: 369433:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [13427.246526] LustreError: 358523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854546 with bad export cookie 8157616940769331055 [13427.247784] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13427.261425] LustreError: 358523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13429.731256] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13429.743343] Lustre: Skipped 1 previous similar message [13429.751433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13429.755683] Lustre: Skipped 2 previous similar messages [13433.584607] Lustre: server umount lustre-MDT0001 complete [13437.362354] Lustre: server umount lustre-OST0000 complete [13447.458327] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [13458.089234] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13495.252341] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 14:16:54 (1787854614) [13502.206477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13502.395706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13502.416318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13502.489336] Lustre: lustre-MDT0000: new disk, initializing [13502.578016] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13502.600606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13506.829538] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13516.086909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13516.229852] Lustre: 382125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13516.276077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13516.279352] Lustre: Skipped 1 previous similar message [13516.330210] Lustre: lustre-MDT0001: new disk, initializing [13516.400392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13516.442911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13516.452573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13520.626600] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13530.208122] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13537.623706] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13544.863595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13545.142866] Lustre: lustre-OST0000: new disk, initializing [13545.148687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13545.157708] Lustre: 384092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13545.267145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13552.208236] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13555.321623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13555.340409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13555.547515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13561.482816] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13570.027388] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13570.035716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13570.044589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13570.047932] Lustre: Skipped 1 previous similar message [13570.422750] Lustre: server umount lustre-MDT0000 complete [13571.052642] LustreError: 382137:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13571.064679] LustreError: 382137:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13574.086190] LustreError: 382930:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13574.087111] LustreError: 384091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854693 with bad export cookie 8157616940769333001 [13574.090942] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13574.102088] LustreError: 382930:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002195eb9a x1874700522528768/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13574.108587] LustreError: 384091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13574.539425] Lustre: server umount lustre-MDT0001 complete [13587.929775] Lustre: server umount lustre-OST0000 complete [13616.423249] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [13627.002546] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13655.504779] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13301 sec ========= 14:19:34 (1787854774) [13657.420526] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:19:36 (1787854776) === [13660.859731] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:19:39 (1787854779) === [13697.588246] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13700.553944] Key type lgssc unregistered [13700.810889] LNet: 390457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13700.818429] LNetError: 390457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13700.829228] LNet: Removed LNI 192.168.201.106@tcp [13701.874445] Key type .llcrypt unregistered [13701.877105] Key type ._llcrypt unregistered