[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483183795 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003062] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.006009] kvm-guest: setup PV IPIs [ 0.008962] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011138] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.014057] *** VALIDATE selinux *** [ 0.022658] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027492] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.031436] *** VALIDATE proc *** [ 0.033133] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035232] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037124] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039024] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043261] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.045147] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046655] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047019] ... version: 2 [ 0.047998] ... bit width: 48 [ 0.048015] ... generic registers: 4 [ 0.049008] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051011] ... fixed-purpose events: 3 [ 0.052009] ... event mask: 000000070000000f [ 0.053244] rcu: Hierarchical SRCU implementation. [ 0.055345] smp: Bringing up secondary CPUs ... [ 0.056529] x86: Booting SMP configuration: [ 0.057018] .... node #0, CPUs: #1 #2 #3 [ 0.062021] smp: Brought up 1 node, 4 CPUs [ 0.064023] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.137308] node 0 deferred pages initialised in 70ms [ 0.140191] devtmpfs: initialized [ 0.141283] x86/mm: Memory block size: 128MB [ 0.144885] gcov: version magic: 0x41383552 [ 0.147344] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153312] pinctrl core: initialized pinctrl subsystem [ 0.156161] [ 0.156644] ************************************************************* [ 0.158009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159009] ** ** [ 0.161014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164014] ** ** [ 0.166011] ** This means that this kernel is built to expose internal ** [ 0.167006] ** IOMMU data structures, which may compromise security on ** [ 0.169048] ** your system. ** [ 0.170010] ** ** [ 0.172010] ** If you see this message and you are not debugging the ** [ 0.174013] ** kernel, report this immediately to your vendor! ** [ 0.177017] ** ** [ 0.179013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181016] ************************************************************* [ 0.184712] NET: Registered protocol family 16 [ 0.186482] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197118] cpuidle: using governor menu [ 0.198601] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201764] PCI: Using configuration type 1 for base access [ 0.203094] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210162] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212322] cryptd: max_cpu_qlen set to 1000 [ 0.216172] ACPI: Added _OSI(Module Device) [ 0.218017] ACPI: Added _OSI(Processor Device) [ 0.220016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.227081] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233502] ACPI: Interpreter enabled [ 0.236068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.238015] ACPI: Using IOAPIC for interrupt routing [ 0.240167] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245613] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260247] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.262032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.265020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273891] acpiphp: Slot [2] registered [ 0.275106] acpiphp: Slot [5] registered [ 0.276074] acpiphp: Slot [6] registered [ 0.278176] acpiphp: Slot [7] registered [ 0.279099] acpiphp: Slot [8] registered [ 0.280068] acpiphp: Slot [9] registered [ 0.281053] acpiphp: Slot [10] registered [ 0.282031] acpiphp: Slot [3] registered [ 0.282972] acpiphp: Slot [4] registered [ 0.284146] acpiphp: Slot [11] registered [ 0.286093] acpiphp: Slot [12] registered [ 0.287161] acpiphp: Slot [13] registered [ 0.289076] acpiphp: Slot [14] registered [ 0.290057] acpiphp: Slot [15] registered [ 0.291080] acpiphp: Slot [16] registered [ 0.293118] acpiphp: Slot [17] registered [ 0.294092] acpiphp: Slot [18] registered [ 0.296108] acpiphp: Slot [19] registered [ 0.298170] acpiphp: Slot [20] registered [ 0.299129] acpiphp: Slot [21] registered [ 0.300059] acpiphp: Slot [22] registered [ 0.302086] acpiphp: Slot [23] registered [ 0.303105] acpiphp: Slot [24] registered [ 0.305098] acpiphp: Slot [25] registered [ 0.306370] acpiphp: Slot [26] registered [ 0.309187] acpiphp: Slot [27] registered [ 0.310073] acpiphp: Slot [28] registered [ 0.311080] acpiphp: Slot [29] registered [ 0.313088] acpiphp: Slot [30] registered [ 0.315095] acpiphp: Slot [31] registered [ 0.316078] PCI host bridge to bus 0000:00 [ 0.317018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.324023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.326022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.329028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.331156] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334066] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338406] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.347741] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365581] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368950] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.372055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375506] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.381016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.394019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.400015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.406201] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.422016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454245] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489136] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.495014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.501014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.524014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.532696] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.539015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.548043] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.570038] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.583467] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.589011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.594013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.615015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.624657] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.632013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.638016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.651922] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.661228] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.664360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.666337] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.668354] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.670218] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676102] iommu: Default domain type: Passthrough [ 0.678421] SCSI subsystem initialized [ 0.680123] ACPI: bus type USB registered [ 0.681121] usbcore: registered new interface driver usbfs [ 0.683071] usbcore: registered new interface driver hub [ 0.685078] usbcore: registered new device driver usb [ 0.686145] pps_core: LinuxPPS API ver. 1 registered [ 0.688011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.691083] PTP clock support registered [ 0.693136] EDAC MC: Ver: 3.0.0 [ 0.695144] PCI: Using ACPI for IRQ routing [ 0.697906] NetLabel: Initializing [ 0.699011] NetLabel: domain hash size = 128 [ 0.701014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.703077] NetLabel: unlabeled traffic allowed by default [ 0.705118] vgaarb: loaded [ 0.707318] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.709013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.715317] clocksource: Switched to clocksource kvm-clock [ 0.818449] VFS: Disk quotas dquot_6.6.0 [ 0.820162] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.822842] *** VALIDATE ramfs *** [ 0.824203] *** VALIDATE hugetlbfs *** [ 0.825727] pnp: PnP ACPI init [ 0.828155] pnp: PnP ACPI: found 6 devices [ 0.849586] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.852598] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.854266] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.856263] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.858289] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.861026] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.864025] NET: Registered protocol family 2 [ 0.866419] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.871372] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.875268] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.881212] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.885107] TCP: Hash tables configured (established 65536 bind 65536) [ 0.888236] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.891356] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.894518] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.897622] NET: Registered protocol family 1 [ 0.900262] RPC: Registered named UNIX socket transport module. [ 0.902600] RPC: Registered udp transport module. [ 0.905597] RPC: Registered tcp transport module. [ 0.907365] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.909820] NET: Registered protocol family 44 [ 0.911709] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.914151] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916478] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.919156] PCI: CLS 0 bytes, default 64 [ 0.920807] Unpacking initramfs... [ 2.311975] debug: unmapping init [mem 0xffff9a9cbcc54000-0xffff9a9cbffbffff] [ 2.315696] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.318399] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.321964] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.800936] Initialise system trusted keyrings [ 2.802570] Key type blacklist registered [ 2.804930] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.813256] zbud: loaded [ 2.816210] *** VALIDATE nfs *** [ 2.817481] *** VALIDATE nfs4 *** [ 2.819132] pstore: using deflate compression [ 2.823892] Platform Keyring initialized [ 2.923485] NET: Registered protocol family 38 [ 2.925629] Key type asymmetric registered [ 2.927596] Asymmetric key parser 'x509' registered [ 2.929080] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.931120] io scheduler mq-deadline registered [ 2.932539] io scheduler kyber registered [ 2.933937] io scheduler bfq registered [ 2.935071] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.937564] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.940875] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.944400] ACPI: Power Button [PWRF] [ 2.950095] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.956805] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.968798] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.974665] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.989993] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.017799] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.047617] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.052930] Non-volatile memory driver v1.3 [ 3.054138] Linux agpgart interface v0.103 [ 3.083421] virtio_blk virtio1: [vda] 150608 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.085680] vda: detected capacity change from 0 to 77111296 [ 3.104914] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.107243] vdb: detected capacity change from 0 to 1073741824 [ 3.125669] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.128710] vdc: detected capacity change from 0 to 2621440000 [ 3.144047] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.146263] vdd: detected capacity change from 0 to 2621440000 [ 3.167929] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.171055] vde: detected capacity change from 0 to 4294967296 [ 3.183730] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.186653] vdf: detected capacity change from 0 to 4294967296 [ 3.193019] libphy: Fixed MDIO Bus: probed [ 3.198635] usbcore: registered new interface driver usbserial_generic [ 3.200895] usbserial: USB Serial support registered for generic [ 3.202917] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.207169] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.208725] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.211161] mousedev: PS/2 mouse device common for all mice [ 3.213515] rtc_cmos 00:05: RTC can wake from S4 [ 3.217406] rtc_cmos 00:05: registered as rtc0 [ 3.219538] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.219562] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.221604] intel_pstate: CPU model not supported [ 3.227557] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.229766] hid: raw HID events driver (C) Jiri Kosina [ 3.232401] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.233186] usbcore: registered new interface driver usbhid [ 3.236888] usbhid: USB HID core driver [ 3.238020] drop_monitor: Initializing network drop monitor service [ 3.240257] Initializing XFRM netlink socket [ 3.242295] NET: Registered protocol family 10 [ 3.244447] Segment Routing with IPv6 [ 3.245855] NET: Registered protocol family 17 [ 3.247350] mpls_gso: MPLS GSO support [ 3.251987] RAS: Correctable Errors collector initialized. [ 3.253852] AVX version of gcm_enc/dec engaged. [ 3.255298] AES CTR mode by8 optimization enabled [ 3.320808] sched_clock: Marking stable (3320780050, 0)->(4254739227, -933959177) [ 3.324517] registered taskstats version 1 [ 3.326422] Loading compiled-in X.509 certificates [ 3.328933] zswap: loaded using pool lzo/zbud [ 3.352194] Key type big_key registered [ 3.361918] Key type encrypted registered [ 3.363434] ima: No TPM chip found, activating TPM-bypass! [ 3.365404] ima: Allocated hash algorithm: sha1 [ 3.366918] ima: No architecture policies found [ 3.368154] evm: Initialising EVM extended attributes: [ 3.370452] evm: security.selinux [ 3.371695] evm: security.ima [ 3.372717] evm: security.capability [ 3.374322] evm: HMAC attrs: 0x1 [ 3.376431] rtc_cmos 00:05: setting system clock to 2026-09-29 19:35:41 UTC (1790710541) [ 3.381720] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 3.385015] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 3.394273] Write protecting the kernel read-only data: 28672k [ 3.397339] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.400133] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 3.432578] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.440919] systemd[1]: Detected virtualization kvm. [ 3.442695] systemd[1]: Detected architecture x86-64. [ 3.444597] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.472855] systemd[1]: No hostname configured. [ 3.474353] systemd[1]: Set hostname to . [ 3.476832] random: systemd: uninitialized urandom read (16 bytes read) [ 3.479798] systemd[1]: Initializing machine ID from random generator. [ 3.591859] random: systemd: uninitialized urandom read (16 bytes read) [ 3.594587] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.599339] random: systemd: uninitialized urandom read (16 bytes read) [ 3.601741] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.606482] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.170767] device-mapper: uevent: version 1.0.3 [ 4.173430] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.820646] virtio_net virtio0 ens2: renamed from eth0 [ 4.824954] random: fast init done [ 4.913272] scsi host0: ata_piix [ 4.959945] scsi host1: ata_piix [ 4.961476] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.964233] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.176713] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.880181] random: crng init done [ 9.881407] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.204290] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.416083] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.702464] SELinux: Disabled at runtime. [ 11.762570] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.773453] systemd[1]: Detected virtualization kvm. [ 11.775732] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.318139] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.322304] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.328487] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.334358] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.338120] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.346507] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.357398] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.502550] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.896603] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.244731] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.261072] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.432792] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.452710] EDAC sbridge: Ver: 1.1.2 [ 15.031400] Key type dns_resolver registered [ 15.373977] NFS: Registering the id_resolver key type [ 15.376528] Key type id_resolver registered [ 15.378380] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg322-server login: [ 73.976675] libcfs: loading out-of-tree module taints kernel. [ 74.032258] Key type ._llcrypt registered [ 74.035417] Key type .llcrypt registered [ 74.199756] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 95.850486] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 97.648473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.675596] alg: No test for adler32 (adler32-zlib) [ 99.131234] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 100.061623] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 101.944194] Key type lgssc registered [ 104.147811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 126.244570] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.119045] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 188.547301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 188.603134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.891495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 189.919768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 190.030298] Lustre: lustre-MDT0000: new disk, initializing [ 190.121296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.137932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.991776] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.392323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.502659] Lustre: 6502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.543899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.551474] Lustre: Skipped 1 previous similar message [ 210.658205] Lustre: lustre-MDT0001: new disk, initializing [ 210.742956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.769819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 210.791735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.220457] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.015448] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 230.488958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 230.737792] Lustre: lustre-OST0000: new disk, initializing [ 230.741332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 230.747951] Lustre: 8443:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.812723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.116141] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.651175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.666849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.738387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 252.289570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.432917] Lustre: lustre-OST0001: new disk, initializing [ 252.436768] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 252.442407] Lustre: 9515:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.514449] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.267448] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.641309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.657159] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.742697] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 271.921417] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.548948] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 286.959419] hrtimer: interrupt took 3409722 ns [ 287.795439] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.985148] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing yml_node [ 299.207083] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 302.985777] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 306.506378] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 308.734828] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 29 15:40:45 EDT 2026 [ 330.187340] Lustre: DEBUG MARKER: only running tests: 76-120 [ 331.964629] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 333.674540] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 352.737995] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.747152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.755333] Lustre: Skipped 1 previous similar message [ 352.778811] Lustre: Skipped 3 previous similar messages [ 356.902774] Lustre: server umount lustre-MDT0000 complete [ 360.479758] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790710898 with bad export cookie 254673517351008131 [ 360.481707] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.494159] LustreError: 6496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 362.977098] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.991959] LustreError: 6511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.998448] Lustre: Skipped 2 previous similar messages [ 362.999160] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 363.055865] LustreError: 6511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 365.984917] LustreError: 6509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.985030] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.030888] Lustre: Skipped 2 previous similar messages [ 366.831546] Lustre: server umount lustre-MDT0001 complete [ 371.837469] Lustre: server umount lustre-OST0000 complete [ 375.935693] Lustre: server umount lustre-OST0001 complete [ 383.414781] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 393.843870] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 433.857021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.227419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 434.285398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 434.394515] Lustre: lustre-MDT0000: new disk, initializing [ 434.456391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.476823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 439.076221] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 449.591488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.701338] Lustre: 15116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 449.795084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 449.802978] Lustre: Skipped 1 previous similar message [ 449.884710] Lustre: lustre-MDT0001: new disk, initializing [ 449.988809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 450.049601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 450.076852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 453.937737] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 463.251689] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 468.696982] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 475.615947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 475.920691] Lustre: lustre-OST0000: new disk, initializing [ 475.925472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 475.944295] Lustre: 17086:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 476.044805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.437499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 477.449027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 477.546786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 481.906401] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 491.085858] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 496.326900] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 496.544938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 501.221874] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 501.455503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 508.385948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 508.386857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.399721] LustreError: Skipped 1 previous similar message [ 508.403182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.413631] Lustre: Skipped 2 previous similar messages [ 508.443469] Lustre: Skipped 1 previous similar message [ 509.692346] Lustre: server umount lustre-OST0000 complete [ 516.577708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 516.592473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.612468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 519.971910] Lustre: server umount lustre-MDT0000 complete [ 523.763531] LustreError: 15129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.776456] LustreError: 15129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 524.173583] LustreError: 17097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711062 with bad export cookie 254673517351009657 [ 524.182563] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 524.187683] LustreError: 17097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 524.591408] Lustre: server umount lustre-MDT0001 complete [ 530.855757] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 15:44:27 (1790711067) [ 538.732847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.357988] LustreError: 19353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.457111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.216907] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 544.745660] LustreError: 19354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.664831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.179987] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 568.699058] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 574.863865] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 582.180303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 588.888726] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 597.241261] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 604.356783] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 614.398215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.406733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 614.424538] Lustre: Skipped 2 previous similar messages [ 614.441855] Lustre: Skipped 4 previous similar messages [ 616.686705] Lustre: server umount lustre-MDT0000 complete [ 619.493631] LustreError: 20332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.520398] LustreError: 20332:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 620.700383] LustreError: 19335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711158 with bad export cookie 254673517351010546 [ 620.702455] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 620.715350] LustreError: 19335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 621.415627] Lustre: server umount lustre-MDT0001 complete [ 636.258590] Lustre: server umount lustre-OST0000 complete [ 659.678918] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 671.961844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.682342] LustreError: 23740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.789557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 672.796667] Lustre: Skipped 2 previous similar messages [ 678.407832] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 689.022977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.546546] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 697.694930] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 706.048539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 715.770446] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 727.604712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 727.853180] Lustre: lustre-OST0001: new disk, initializing [ 727.858617] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 727.874930] Lustre: 26059:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 729.927938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 729.943816] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 730.135298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 736.809336] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 747.803895] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 752.025755] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 771.043109] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.052114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 771.056136] Lustre: Skipped 3 previous similar messages [ 771.069744] Lustre: Skipped 3 previous similar messages [ 773.349190] Lustre: server umount lustre-MDT0000 complete [ 776.168290] LustreError: 23740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 776.189051] LustreError: 23740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 777.494996] LustreError: 24881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711315 with bad export cookie 254673517351011393 [ 777.496786] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 777.532152] LustreError: 24881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 778.004453] Lustre: server umount lustre-MDT0001 complete [ 792.677621] Lustre: server umount lustre-OST0000 complete [ 806.963552] Lustre: server umount lustre-OST0001 complete [ 825.708421] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 836.365559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.921379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 836.927940] Lustre: Skipped 3 previous similar messages [ 841.685828] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 842.211238] LustreError: 29080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.243908] LustreError: 29080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 850.826856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.057168] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 859.342932] Lustre: 30227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 867.206732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 874.571110] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 884.754663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 893.348585] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 902.698856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 906.775493] Lustre: 32107:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 915.941878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.960047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 915.964344] Lustre: Skipped 3 previous similar messages [ 915.986361] Lustre: Skipped 2 previous similar messages [ 919.806808] Lustre: server umount lustre-MDT0000 complete [ 924.628773] LustreError: 29061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711462 with bad export cookie 254673517351012688 [ 924.630725] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 924.651675] LustreError: 29061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 925.047521] Lustre: server umount lustre-MDT0001 complete [ 939.757671] Lustre: server umount lustre-OST0000 complete [ 942.625369] Lustre: 3633:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790711464/real 1790711464] req@ffff9a9c036e7480 x1877696198258944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790711480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 943.976449] Lustre: server umount lustre-OST0001 complete [ 953.069803] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 15:51:30 (1790711490) [ 987.467734] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 998.568488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.150296] LustreError: 34912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.183917] LustreError: 34912:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 1004.426473] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1014.390078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.858076] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1024.962678] Lustre: 36062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1032.685310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1040.909986] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1050.837934] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1058.251379] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1066.600653] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1070.764553] Lustre: 37942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1076.713618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.718810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1076.722313] Lustre: Skipped 3 previous similar messages [ 1076.730226] Lustre: Skipped 2 previous similar messages [ 1081.313706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1083.161604] Lustre: server umount lustre-MDT0000 complete [ 1088.100853] LustreError: 34892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711626 with bad export cookie 254673517351013794 [ 1088.104343] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1088.111015] LustreError: 34892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1088.734981] Lustre: server umount lustre-MDT0001 complete [ 1103.978390] Lustre: server umount lustre-OST0000 complete [ 1107.936119] Lustre: 3635:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790711630/real 1790711630] req@ffff9a9d01324e00 x1877696198311296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790711646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1108.683816] Lustre: server umount lustre-OST0001 complete [ 1118.657823] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 15:54:15 (1790711655) [ 1136.851270] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1149.183042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.861882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.873643] Lustre: Skipped 7 previous similar messages [ 1155.403621] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1165.334079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.683574] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1175.100835] Lustre: 41234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1183.020666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1190.940222] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1201.647771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1209.613135] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1218.522546] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1222.796882] Lustre: 43110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1223.939936] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1223.944649] Lustre: Skipped 1 previous similar message [ 1227.237878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1227.242696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1227.254079] Lustre: Skipped 2 previous similar messages [ 1227.261506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1227.264487] Lustre: Skipped 4 previous similar messages [ 1233.880864] Lustre: server umount lustre-MDT0000 complete [ 1238.114499] LustreError: 40064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711776 with bad export cookie 254673517351014900 [ 1238.118273] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1238.123575] LustreError: 40064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1244.594184] Lustre: server umount lustre-MDT0001 complete [ 1254.892698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.817225] LustreError: 41597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1264.841939] LustreError: 41597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 30 previous similar messages [ 1270.745148] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1273.216493] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1280.405466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.146130] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1286.157407] Lustre: Skipped 1 previous similar message [ 1286.177875] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1294.987490] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1297.000580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1303.688844] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1305.473101] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1311.738583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1317.332130] Lustre: server umount lustre-MDT0000 complete [ 1321.169266] LustreError: 40066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790711859 with bad export cookie 254673517351015999 [ 1321.175861] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1321.183426] LustreError: 40066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1321.563724] Lustre: server umount lustre-MDT0001 complete [ 1335.841371] Lustre: server umount lustre-OST0000 complete [ 1338.336746] Lustre: 3635:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790711860/real 1790711860] req@ffff9a9c0378ce00 x1877696198404480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790711876 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1339.668786] Lustre: server umount lustre-OST0001 complete [ 1350.900510] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 15:58:07 (1790711887) [ 1371.809324] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1383.028115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.273901] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1397.921967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.877741] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1407.321847] Lustre: 48874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1414.052197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1422.027801] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1435.172419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1444.880950] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1453.630180] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1459.770759] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1475.043129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1475.057674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.083120] Lustre: Skipped 10 previous similar messages [ 1480.965308] Lustre: server umount lustre-MDT0000 complete [ 1485.190779] LustreError: 47704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790712023 with bad export cookie 254673517351016818 [ 1485.208315] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1485.212599] LustreError: 47704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1485.726082] Lustre: server umount lustre-MDT0001 complete [ 1501.060859] Lustre: server umount lustre-OST0000 complete [ 1501.984148] Lustre: 3635:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790712024/real 1790712024] req@ffff9a9d048f9180 x1877696198467584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790712040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1506.016954] Lustre: server umount lustre-OST0001 complete [ 1518.909425] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 16:00:55 (1790712055) [ 1523.135843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1525.813286] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 16:01:02 (1790712062) [ 1539.520206] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 1553.535336] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1583.134443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1583.566431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1583.755647] Lustre: lustre-MDT0000: new disk, initializing [ 1583.930752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1591.410050] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1603.111784] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1611.888579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1612.368558] Lustre: lustre-OST0000: new disk, initializing [ 1612.382762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1612.405246] Lustre: Skipped 1 previous similar message [ 1612.417707] Lustre: 54966:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1613.549797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1613.561440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1613.633158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1619.926237] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1631.231540] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1634.133640] Lustre: DEBUG MARKER: create test files [ 1675.237788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.249294] Lustre: Skipped 28 previous similar messages [ 1678.722906] Lustre: server umount lustre-OST0000 complete [ 1683.687092] Lustre: server umount lustre-MDT0000 complete [ 1696.878528] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1700.086583] Key type lgssc unregistered [ 1700.383804] LNet: 56599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.392604] LNetError: 56599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.423532] LNet: Removed LNI 192.168.203.122@tcp [ 1701.476289] Key type .llcrypt unregistered [ 1701.479745] Key type ._llcrypt unregistered [ 1734.462897] Key type ._llcrypt registered [ 1734.465724] Key type .llcrypt registered [ 1734.555582] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1735.619237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1735.710062] alg: No test for adler32 (adler32-zlib) [ 1736.965517] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 1737.281467] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1738.920242] Key type lgssc registered [ 1740.007843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1748.232844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1748.260720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.661164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.827460] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1763.122064] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1770.591764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1770.958517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.069953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1774.152575] Lustre: 59350:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1774.174868] Lustre: 59350:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1774.196607] Lustre: 59350:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1774.216788] Lustre: 59350:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1774.236416] Lustre: 59350:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1774.246653] Lustre: 59350:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1777.310526] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1786.105729] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1792.172595] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1798.090915] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1809.893627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1809.894841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.932139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1812.353561] Lustre: server umount lustre-OST0000 complete [ 1818.473331] Lustre: server umount lustre-MDT0000 complete [ 1833.485744] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1836.558724] Key type lgssc unregistered [ 1836.917567] LNet: 60959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.927988] LNetError: 60959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.955925] LNet: Removed LNI 192.168.203.122@tcp [ 1838.026806] Key type .llcrypt unregistered [ 1838.028636] Key type ._llcrypt unregistered [ 1872.396694] Key type ._llcrypt registered [ 1872.399675] Key type .llcrypt registered [ 1872.535481] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 1873.792974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1873.858084] alg: No test for adler32 (adler32-zlib) [ 1874.919793] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 1875.177465] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 1876.856275] Key type lgssc registered [ 1878.398475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1887.018991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1887.055436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.564309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.899493] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1903.302561] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1913.216433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1913.864292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.907681] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 1931.627429] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1933.309851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1933.421638] Lustre: 63811:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 1933.432259] Lustre: 63811:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 32/128/1, destroy: 0/0/0 [ 1933.442494] Lustre: 63811:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1933.455334] Lustre: 63811:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1933.467095] Lustre: 63811:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1933.484893] Lustre: 63811:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1943.967655] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1953.768634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.793900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.818961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1958.883947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1959.347786] Lustre: server umount lustre-OST0000 complete [ 1964.047152] Lustre: server umount lustre-MDT0000 complete [ 1976.467397] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 1979.273466] Key type lgssc unregistered [ 1979.568454] LNet: 65388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.590822] LNetError: 65388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1979.606953] LNet: Removed LNI 192.168.203.122@tcp [ 1980.556085] Key type .llcrypt unregistered [ 1980.559802] Key type ._llcrypt unregistered [ 2000.655912] Key type ._llcrypt registered [ 2000.658153] Key type .llcrypt registered [ 2000.759319] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 2017.240948] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2018.115151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2018.142050] alg: No test for adler32 (adler32-zlib) [ 2019.150691] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 2019.387150] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 2021.096212] Key type lgssc registered [ 2022.184222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2067.888481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2067.955529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.206204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2069.238873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2069.292482] Lustre: lustre-MDT0000: new disk, initializing [ 2069.336814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.357311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2074.671466] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2086.447027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.581173] Lustre: 68560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2086.639937] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2086.647322] Lustre: Skipped 1 previous similar message [ 2086.711434] Lustre: lustre-MDT0001: new disk, initializing [ 2086.825980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2086.865446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2086.887870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2091.958555] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2102.636597] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2108.573955] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2116.136513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2116.409578] Lustre: lustre-OST0000: new disk, initializing [ 2116.413182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2116.419457] Lustre: 70527:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2116.488639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.870696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2122.909140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2123.083297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2124.657183] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2137.279860] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2142.748496] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2143.073612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2147.766417] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2148.015114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2153.953970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2153.962779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.975541] LustreError: Skipped 1 previous similar message [ 2153.988282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2153.998718] Lustre: Skipped 1 previous similar message [ 2154.010939] Lustre: Skipped 1 previous similar message [ 2156.669255] Lustre: server umount lustre-OST0000 complete [ 2164.201903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.215041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.224809] Lustre: Skipped 1 previous similar message [ 2164.249124] Lustre: Skipped 1 previous similar message [ 2169.314591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.327305] Lustre: Skipped 1 previous similar message [ 2170.392884] Lustre: server umount lustre-MDT0000 complete [ 2176.084809] LustreError: 68550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790712714 with bad export cookie 10308426844084756381 [ 2176.089930] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2176.098139] LustreError: 68550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2176.751872] Lustre: server umount lustre-MDT0001 complete [ 2187.857585] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 16:12:04 (1790712724) [ 2200.546488] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2243.854089] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 2253.628510] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2292.206579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.457940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2292.502796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2292.634716] Lustre: lustre-MDT0000: new disk, initializing [ 2292.714749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.744238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2297.289399] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2307.718655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.831779] Lustre: 76556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2307.942081] LustreError: 76565:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.961334] LustreError: 76565:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2307.968830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2307.976918] Lustre: Skipped 1 previous similar message [ 2308.094078] Lustre: lustre-MDT0001: new disk, initializing [ 2308.212155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2308.243694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2308.259067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2311.927634] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2321.662744] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2328.199935] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2336.116538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2336.471255] Lustre: lustre-OST0000: new disk, initializing [ 2336.483767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2336.499915] Lustre: 78525:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2336.637737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.184171] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2343.545158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2343.550059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2343.656379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2354.031196] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2359.674081] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2359.997487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2366.570417] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2366.909697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2374.633430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2374.640474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.647752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2374.669556] Lustre: Skipped 1 previous similar message [ 2374.685764] Lustre: Skipped 1 previous similar message [ 2375.823499] Lustre: server umount lustre-OST0000 complete [ 2384.865306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2384.869726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.874347] LustreError: Skipped 1 previous similar message [ 2384.875216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2384.906337] Lustre: Skipped 1 previous similar message [ 2386.373204] Lustre: server umount lustre-MDT0000 complete [ 2389.986569] LustreError: 77361:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2390.006670] LustreError: 77361:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2390.238979] LustreError: 78524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790712928 with bad export cookie 10308426844084757263 [ 2390.244770] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.252283] LustreError: 78524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2390.532680] Lustre: server umount lustre-MDT0001 complete [ 2400.035724] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 16:15:37 (1790712937) [ 2407.053900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.486784] LustreError: 80937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.553465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.148491] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2413.025386] LustreError: 80938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2417.133908] LustreError: 80937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.442885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2424.845636] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2432.728614] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2438.983570] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2446.247202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.638204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.642420] Lustre: Skipped 1 previous similar message [ 2453.024677] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2461.117256] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2465.063581] Lustre: 83607:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting 69f95234-45cc-4afd-bb40-38219770c1ee at adminstrative request [ 2470.380535] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2470.396146] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2480.624149] Lustre: MGS: Client 69f95234-45cc-4afd-bb40-38219770c1ee (at 0@lo) reconnecting [ 2480.634778] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x8f0ee3992ac5e67a to 0x8f0ee3992ac5e921 [ 2480.653425] Lustre: MGC192.168.203.122@tcp: Connection restored to 0@lo (at 0@lo) [ 2502.909944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2503.105324] Lustre: lustre-OST0001: new disk, initializing [ 2503.111302] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2503.120228] Lustre: 84000:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2508.961790] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2513.019459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2513.030218] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2513.132929] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2518.231319] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2523.620533] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2523.631432] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2523.632582] LustreError: Skipped 1 previous similar message [ 2523.635150] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2523.641666] Lustre: Skipped 1 previous similar message [ 2523.650358] Lustre: Skipped 1 previous similar message [ 2535.392121] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2535.472266] Lustre: server umount lustre-OST0001 complete [ 2538.979874] LustreError: 84855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.995144] LustreError: 84855:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2540.009228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2540.018156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2540.046544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2540.057562] Lustre: Skipped 5 previous similar messages [ 2554.336126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2554.696313] Lustre: server umount lustre-OST0000 complete [ 2559.467544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.498366] Lustre: Skipped 2 previous similar messages [ 2565.658975] Lustre: server umount lustre-MDT0000 complete [ 2569.582891] LustreError: 80917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790713107 with bad export cookie 10308426844084758817 [ 2569.588557] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2569.596544] LustreError: 80917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2570.096743] Lustre: server umount lustre-MDT0001 complete [ 2579.383942] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 16:18:36 (1790713116) [ 2581.120744] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2582.936660] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 16:18:39 (1790713119) [ 2584.377193] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2586.472254] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 16:18:43 (1790713123) [ 2588.239934] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2590.073544] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 16:18:47 (1790713127) [ 2591.941373] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2593.492861] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 16:18:50 (1790713130) [ 2595.106244] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2600.871715] loop: module loaded [ 2600.897334] loop0: detected capacity change from 0 to 4294967296 [ 2600.968597] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2602.330351] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2605.776460] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2609.791497] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2637.249610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2637.312157] LustreError: 88126:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2637.325800] LustreError: 88126:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2637.333664] LustreError: 88126:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2637.343405] LustreError: 88126:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2657.416525] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 2668.680836] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 2705.139365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2705.420211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2705.453582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2705.583155] Lustre: lustre-MDT0000: new disk, initializing [ 2705.686325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2705.694254] Lustre: Skipped 1 previous similar message [ 2705.709229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2710.151681] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2720.409794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.572652] Lustre: 90993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2720.850658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2724.805993] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2733.366373] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2738.979741] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2746.098203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2746.321477] Lustre: 92959:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2752.029760] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2753.588952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2753.708992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2760.492788] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2765.047723] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2765.223909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2769.868678] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2770.159303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2774.500428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2774.505248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2774.517415] LustreError: Skipped 2 previous similar messages [ 2774.524983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2774.524990] Lustre: Skipped 11 previous similar messages [ 2778.503579] Lustre: server umount lustre-OST0000 complete [ 2789.167966] Lustre: server umount lustre-MDT0000 complete [ 2789.859763] LustreError: 91824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.886872] LustreError: 91824:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2793.836108] LustreError: 90984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790713331 with bad export cookie 10308426844084759433 [ 2793.865159] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2794.243203] Lustre: server umount lustre-MDT0001 complete [ 2804.867370] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 16:22:21 (1790713341) [ 2813.001563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.219325] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2823.155152] LustreError: 95370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.172674] LustreError: 95370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2827.315949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.274404] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2840.406465] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2846.926366] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2854.258819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.658055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.667388] Lustre: Skipped 4 previous similar messages [ 2861.688572] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2869.575748] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2873.985604] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2874.177843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2878.610786] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2878.793094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2885.375793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2885.554075] Lustre: lustre-OST0001: new disk, initializing [ 2885.561417] Lustre: Skipped 2 previous similar messages [ 2885.569963] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2885.574848] Lustre: Skipped 3 previous similar messages [ 2885.579972] Lustre: 98740:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2892.324853] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2895.972574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2895.992335] Lustre: Skipped 2 previous similar messages [ 2895.999100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2896.091322] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2901.140294] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2906.036373] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2906.202931] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2910.823726] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2911.032726] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2930.109632] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2956.745397] Lustre: Failing over lustre-MDT0000 [ 2957.093607] Lustre: server umount lustre-MDT0000 complete [ 2957.798522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.806732] LustreError: 95366:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.816766] Lustre: Skipped 6 previous similar messages [ 2957.851359] LustreError: 95366:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2973.090938] Lustre: 66769:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790713495/real 1790713495] req@00000000efe9c5d1 x1877698210448768/t0(0) o400->MGC192.168.203.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790713511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2973.160192] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.128265] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2983.132785] LDISKFS-fs (dm-0): recovery complete [ 2983.141688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.403282] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8f0ee3992ac6b395 [ 2983.416372] Lustre: MGC192.168.203.122@tcp: Connection restored to 0@lo (at 0@lo) [ 2983.753323] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2984.896204] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2984.897208] Lustre: 97211:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2987.866864] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 2989.055282] Lustre: 95370:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2989.072394] Lustre: 95370:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 2989.080526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2989.172273] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2989.499854] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2989.828992] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2989.844174] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2990.168144] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2990.179771] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2990.187794] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2990.198826] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 7 previous similar messages [ 2990.833432] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2990.849290] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2991.184235] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2991.190957] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 2 previous similar messages [ 2992.483187] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2992.514248] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 6 previous similar messages [ 2992.842801] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2992.862238] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 5 previous similar messages [ 2993.185259] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2993.200802] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 5 previous similar messages [ 2996.705267] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2996.731856] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 2997.078258] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2997.093396] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 2997.424137] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2997.433377] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 3004.746391] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3004.765178] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 3005.089540] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3005.106593] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 3005.744322] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3005.755551] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 25 previous similar messages [ 3020.814379] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3020.839889] Lustre: 101592:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 49 previous similar messages [ 3021.170898] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3021.185995] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 3021.832366] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3021.838200] LustreError: 101592:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 3044.096462] Lustre: 101592:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3044.110445] Lustre: 101592:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3044.121672] Lustre: 101592:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 77a74bb4-75e9-43d5-9f04-a8f94b0db557@192.168.203.22@tcp [ 3044.136843] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3044.146410] Lustre: 101592:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@000000006f23913d x1877698194508672/t0(8589934768) o35->77a74bb4-75e9-43d5-9f04-a8f94b0db557@192.168.203.22@tcp:62/0 lens 392/0 e 0 to 0 dl 1790713592 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3044.178674] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3044.197399] Lustre: lustre-MDT0000: Denying connection for new client 77a74bb4-75e9-43d5-9f04-a8f94b0db557 (at 192.168.203.22@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3044.214768] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3044.271311] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3044.281848] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3044.283386] Lustre: Skipped 2 previous similar messages [ 3044.359921] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 3044.362522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:77 to 0x280000401:577) [ 3056.613108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3056.632943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3056.636324] Lustre: Skipped 3 previous similar messages [ 3062.404484] Lustre: server umount lustre-OST0000 complete [ 3073.325945] Lustre: server umount lustre-OST0001 complete [ 3083.964118] Lustre: server umount lustre-MDT0000 complete [ 3086.317826] LustreError: 95371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.350044] LustreError: 95371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 35 previous similar messages [ 3088.544170] LustreError: 95350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790713626 with bad export cookie 10308426844084810645 [ 3088.554621] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3088.561020] LustreError: 95350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3089.231984] Lustre: server umount lustre-MDT0001 complete [ 3101.118818] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 16:27:18 (1790713638) [ 3109.041104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3109.046956] Lustre: *** cfs_fail_loc=197, val=0*** [ 3124.641905] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3129.764092] LustreError: 103566:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 3134.905401] Lustre: *** cfs_fail_loc=197, val=0*** [ 3134.910891] Lustre: Skipped 65 previous similar messages [ 3155.488585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.496648] Lustre: Skipped 2 previous similar messages [ 3160.676792] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3167.768525] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3169.735992] Lustre: server umount lustre-OST0000 complete [ 3179.032734] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 16:28:35 (1790713715) [ 3200.597464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.604722] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3215.482945] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3219.547296] LustreError: 105216:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3219.561289] LustreError: 105216:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3219.575318] LustreError: 105216:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 3220.002552] Lustre: server umount lustre-MDT0000 complete [ 3252.091442] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3261.967823] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3296.069138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.382717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3296.420492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3296.506103] Lustre: lustre-MDT0000: new disk, initializing [ 3296.603567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3300.995692] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3311.478880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.573861] Lustre: 109393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3311.795341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3316.210108] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3325.752090] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3331.724838] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3338.470659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3338.752825] Lustre: 111361:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3345.430575] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3347.680674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3355.421304] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3359.595247] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3359.811886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3363.502419] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3363.694326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3368.421447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3368.421596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.433637] LustreError: Skipped 5 previous similar messages [ 3368.434412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3368.454936] Lustre: Skipped 7 previous similar messages [ 3368.474423] Lustre: Skipped 8 previous similar messages [ 3371.662926] Lustre: server umount lustre-OST0000 complete [ 3382.253515] Lustre: server umount lustre-MDT0000 complete [ 3383.781902] LustreError: 109405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.799774] LustreError: 109405:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3385.800773] LustreError: 109385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790713923 with bad export cookie 10308426844084821229 [ 3385.805669] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3385.812838] LustreError: 109385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3386.304924] Lustre: server umount lustre-MDT0001 complete [ 3394.135291] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 16:32:11 (1790713931) [ 3401.773908] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3404.509412] Key type lgssc unregistered [ 3404.756674] LNet: 113848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3404.771325] LNetError: 113848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3404.800331] LNet: Removed LNI 192.168.203.122@tcp [ 3405.615295] Key type .llcrypt unregistered [ 3405.618986] Key type ._llcrypt unregistered [ 3423.067534] Key type ._llcrypt registered [ 3423.071873] Key type .llcrypt registered [ 3423.169606] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3437.481884] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3438.613806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3438.771326] alg: No test for adler32 (adler32-zlib) [ 3439.810792] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 3440.092672] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3441.768258] Key type lgssc registered [ 3442.911709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3483.125538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3483.178622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.528480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3484.578586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3484.676178] Lustre: lustre-MDT0000: new disk, initializing [ 3484.760598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.774727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3488.521419] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3497.388152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3497.666433] Lustre: lustre-OST0000: new disk, initializing [ 3497.673168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3497.677529] Lustre: Skipped 1 previous similar message [ 3497.683634] Lustre: 118045:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3497.774163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.715982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3499.723208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3499.747082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3503.510756] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3513.716472] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3513.914974] Lustre: lustre-OST0001: new disk, initializing [ 3513.919501] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3513.925270] Lustre: 118908:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3514.016996] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3520.393097] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3521.142644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3521.151866] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3521.193201] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3531.236804] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3531.262380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.269073] Lustre: Skipped 1 previous similar message [ 3535.000194] Lustre: server umount lustre-MDT0000 complete [ 3540.647270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3544.529913] LustreError: 117232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714082 with bad export cookie 1110377858202204653 [ 3544.538768] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3544.547976] LustreError: 117232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3544.645570] Lustre: server umount lustre-OST0000 complete [ 3548.513610] Lustre: server umount lustre-OST0001 complete [ 3559.435896] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 16:34:55 (1790714095) [ 3584.575696] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 16:35:21 (1790714121) [ 3593.692681] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3614.660251] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 16:35:51 (1790714151) [ 3649.954202] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 3660.603496] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3695.183328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.427209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3695.480421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3695.575395] Lustre: lustre-MDT0000: new disk, initializing [ 3695.663688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.675214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3699.535829] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3708.761894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.841394] Lustre: 126661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3708.880645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3708.884061] Lustre: Skipped 1 previous similar message [ 3708.961485] Lustre: lustre-MDT0001: new disk, initializing [ 3709.044444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3709.076030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3709.090083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3713.019337] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3722.617696] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3728.384491] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3735.454938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3735.747755] Lustre: lustre-OST0000: new disk, initializing [ 3735.755475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3735.763603] Lustre: 128628:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3735.884850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.318264] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3741.735682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3741.741946] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3741.852675] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3750.286489] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3755.399099] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3755.618675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3759.928211] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3760.091597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3762.667219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3762.669134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.672276] LustreError: Skipped 1 previous similar message [ 3762.682290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3762.694232] Lustre: Skipped 2 previous similar messages [ 3767.876263] Lustre: server umount lustre-OST0000 complete [ 3772.897747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.909493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.909847] Lustre: Skipped 1 previous similar message [ 3772.932971] Lustre: Skipped 2 previous similar messages [ 3778.019476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3778.027629] Lustre: Skipped 1 previous similar message [ 3783.146204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3786.208394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3786.424542] Lustre: server umount lustre-MDT0000 complete [ 3788.257893] LustreError: 126667:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.302892] LustreError: 126667:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3790.850769] LustreError: 126653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714328 with bad export cookie 1110377858202205878 [ 3790.855672] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3790.869816] LustreError: 126653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3791.141154] Lustre: server umount lustre-MDT0001 complete [ 3801.771638] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 16:38:58 (1790714338) [ 3809.725326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.260036] LustreError: 131038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.316414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.021279] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3815.395385] LustreError: 131039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3820.529575] LustreError: 131038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3825.124902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.842158] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3837.954718] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3843.151196] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3849.258855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3849.566127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3849.572135] Lustre: Skipped 1 previous similar message [ 3855.149574] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 3862.836441] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3867.460740] Lustre: *** cfs_fail_loc=159, val=0*** [ 3882.983787] Lustre: lustre-MDT0000: Client e676aa08-e0e2-4ef3-8d3f-a523a15f2619 (at 192.168.203.22@tcp) reconnecting [ 3884.981932] Lustre: *** cfs_fail_loc=159, val=0*** [ 3884.999188] Lustre: Skipped 5 previous similar messages [ 3901.429989] Lustre: lustre-MDT0000: Client e676aa08-e0e2-4ef3-8d3f-a523a15f2619 (at 192.168.203.22@tcp) reconnecting [ 3923.425530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.435896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.446311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.454566] Lustre: Skipped 1 previous similar message [ 3923.471906] Lustre: Skipped 2 previous similar messages [ 3924.676236] Lustre: server umount lustre-OST0000 complete [ 3933.167512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3933.183264] LustreError: Skipped 1 previous similar message [ 3933.191428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.884152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3942.899608] Lustre: Skipped 5 previous similar messages [ 3943.392158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3943.754887] Lustre: server umount lustre-MDT0000 complete [ 3948.006652] LustreError: 131039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3948.024225] LustreError: 131039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3948.176724] LustreError: 131018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714486 with bad export cookie 1110377858202206760 [ 3948.178462] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3948.203908] LustreError: 131018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3948.663737] Lustre: server umount lustre-MDT0001 complete [ 3959.945515] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 3962.528142] Key type lgssc unregistered [ 3962.830716] LNet: 134817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3962.842962] LNetError: 134817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3962.865908] LNet: Removed LNI 192.168.203.122@tcp [ 3963.618760] Key type .llcrypt unregistered [ 3963.622251] Key type ._llcrypt unregistered [ 3989.188624] Key type ._llcrypt registered [ 3989.193338] Key type .llcrypt registered [ 3989.292556] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 3990.466533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3990.494919] alg: No test for adler32 (adler32-zlib) [ 3991.595090] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 3991.856338] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 3993.585493] Key type lgssc registered [ 3994.746987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4003.044913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4003.066805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4004.594303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.053952] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4014.058254] LustreError: 136246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4014.085648] LustreError: 136246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4017.342692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4017.803579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4021.982904] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4030.362810] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4037.264978] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4043.502549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4043.861754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4048.936489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4049.925749] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4056.985833] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4070.003826] Lustre: *** cfs_fail_loc=159, val=0*** [ 4086.258388] Lustre: lustre-MDT0000: Client 528428c0-64f2-4c5c-9db7-fb5554ddecf2 (at 192.168.203.22@tcp) reconnecting [ 4088.746874] Lustre: *** cfs_fail_loc=159, val=0*** [ 4105.141219] Lustre: lustre-MDT0001: Client 528428c0-64f2-4c5c-9db7-fb5554ddecf2 (at 192.168.203.22@tcp) reconnecting [ 4106.854234] Lustre: *** cfs_fail_loc=159, val=0*** [ 4106.859617] Lustre: Skipped 4 previous similar messages [ 4123.157044] Lustre: lustre-MDT0001: Client 528428c0-64f2-4c5c-9db7-fb5554ddecf2 (at 192.168.203.22@tcp) reconnecting [ 4139.086312] Lustre: 139306:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4141.584884] Lustre: *** cfs_fail_loc=159, val=0*** [ 4141.586936] Lustre: Skipped 3 previous similar messages [ 4157.932606] Lustre: lustre-MDT0000: Client 13a83764-ead6-4829-af09-067ffaa200eb (at 192.168.203.22@tcp) reconnecting [ 4159.330030] Lustre: *** cfs_fail_loc=159, val=0*** [ 4159.332515] Lustre: Skipped 14 previous similar messages [ 4174.840436] Lustre: lustre-MDT0000: Client 13a83764-ead6-4829-af09-067ffaa200eb (at 192.168.203.22@tcp) reconnecting [ 4176.590807] Lustre: 139596:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4185.058694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.064219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4185.084081] Lustre: Skipped 1 previous similar message [ 4185.110183] Lustre: Skipped 1 previous similar message [ 4189.362451] Lustre: server umount lustre-OST0000 complete [ 4195.299152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.306710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.316926] Lustre: Skipped 1 previous similar message [ 4195.333961] Lustre: Skipped 1 previous similar message [ 4195.908702] Lustre: server umount lustre-MDT0000 complete [ 4199.520420] LustreError: 136225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714737 with bad export cookie 4536300479816632248 [ 4199.531522] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4199.533706] LustreError: 136225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4199.932616] Lustre: server umount lustre-MDT0001 complete [ 4210.191727] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4212.643685] Key type lgssc unregistered [ 4212.919614] LNet: 140647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4212.932793] LNetError: 140647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4212.959150] LNet: Removed LNI 192.168.203.122@tcp [ 4213.792547] Key type .llcrypt unregistered [ 4213.797356] Key type ._llcrypt unregistered [ 4236.215236] Key type ._llcrypt registered [ 4236.217716] Key type .llcrypt registered [ 4236.337521] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4237.200361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4237.222798] alg: No test for adler32 (adler32-zlib) [ 4238.292614] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 4238.570308] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4240.240251] Key type lgssc registered [ 4241.434201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4249.088545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4249.126197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.524860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.930823] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4260.834390] LustreError: 142075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4260.864201] LustreError: 142075:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4263.419509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4263.778630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4268.480188] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4276.750404] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4283.341750] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4291.321088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4291.826098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.931474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4299.452911] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4308.001957] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4308.281705] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4315.339534] Lustre: 144658:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4319.672264] Lustre: 144707:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4321.116888] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4321.936131] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4326.135287] Lustre: 144855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4327.297062] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4328.376964] Lustre: 144954:0:(mdt_handler.c:65:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4334.049367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4334.051612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.060213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.092740] Lustre: Skipped 1 previous similar message [ 4339.184795] LustreError: 143895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.201176] LustreError: 143895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4339.460768] Lustre: server umount lustre-OST0000 complete [ 4344.294184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4344.312172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4344.320450] Lustre: Skipped 1 previous similar message [ 4344.328067] Lustre: Skipped 2 previous similar messages [ 4349.411428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4349.423253] Lustre: Skipped 1 previous similar message [ 4349.808413] Lustre: server umount lustre-MDT0000 complete [ 4354.439392] LustreError: 142055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790714892 with bad export cookie 14224632699308630480 [ 4354.456800] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.465232] LustreError: 142055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4355.000704] Lustre: server umount lustre-MDT0001 complete [ 4366.788728] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4369.973159] Key type lgssc unregistered [ 4370.435881] LNet: 146013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4370.446178] LNetError: 146013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4370.480110] LNet: Removed LNI 192.168.203.122@tcp [ 4371.466918] Key type .llcrypt unregistered [ 4371.469609] Key type ._llcrypt unregistered [ 4398.069466] Key type ._llcrypt registered [ 4398.072132] Key type .llcrypt registered [ 4398.196734] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4399.412930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4399.587081] alg: No test for adler32 (adler32-zlib) [ 4400.710331] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 4400.911958] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4402.568266] Key type lgssc registered [ 4403.698948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4413.179633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4413.203619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4414.698725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.799845] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4424.164643] LustreError: 147444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4424.182456] LustreError: 147444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4429.289267] LustreError: 147445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4430.894603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4431.375676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4437.183804] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4445.859173] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4452.234293] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4459.413533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4459.885181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4467.593990] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4475.654600] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4477.257212] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4484.464603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4492.956430] Lustre: *** cfs_fail_loc=159, val=0*** [ 4508.674131] Lustre: lustre-MDT0000: Client 1dc67746-7e68-4a26-938f-93dfa7c60462 (at 192.168.203.22@tcp) reconnecting [ 4515.298454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.302564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4515.317529] Lustre: Skipped 1 previous similar message [ 4515.327910] Lustre: Skipped 1 previous similar message [ 4520.427404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4520.431699] Lustre: Skipped 1 previous similar message [ 4521.010384] Lustre: server umount lustre-OST0000 complete [ 4528.611312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4528.627586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.651817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.658539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.662332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.790459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.812181] Lustre: Skipped 2 previous similar messages [ 4540.384591] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4540.608913] Lustre: server umount lustre-MDT0000 complete [ 4540.908325] LustreError: 147444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.061831] LustreError: 147445:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.080888] LustreError: 147445:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4545.471112] LustreError: 147425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790715083 with bad export cookie 2827583590858467490 [ 4545.497243] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4546.245575] Lustre: server umount lustre-MDT0001 complete [ 4558.686890] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4561.669700] Key type lgssc unregistered [ 4562.020249] LNet: 151127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4562.029112] LNetError: 151127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4562.054358] LNet: Removed LNI 192.168.203.122@tcp [ 4562.997455] Key type .llcrypt unregistered [ 4563.000504] Key type ._llcrypt unregistered [ 4590.433695] Key type ._llcrypt registered [ 4590.439738] Key type .llcrypt registered [ 4590.597641] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4592.297620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4592.343321] alg: No test for adler32 (adler32-zlib) [ 4593.577491] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 4594.022590] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4595.728279] Key type lgssc registered [ 4597.360452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4606.896213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4606.945763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.760917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.874088] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4618.213691] LustreError: 152559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4618.241348] LustreError: 152559:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4623.341731] LustreError: 152560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4624.915788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4625.490678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4630.813349] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4641.819696] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4651.076212] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4658.965734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4659.325053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.314662] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4670.581299] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4675.837073] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4678.765498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4700.035811] Lustre: lustre-OST0000: evicting 179305c5-6af4-4a2e-84b3-3c642fffcdf4 (at 192.168.203.22@tcp) by administrative request [ 4700.050619] Lustre: Skipped 1 previous similar message [ 4730.342413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.361392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4730.366704] Lustre: Skipped 1 previous similar message [ 4730.367208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4735.488793] LustreError: 154392:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4735.520678] LustreError: 154392:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4735.907948] Lustre: server umount lustre-OST0000 complete [ 4740.582237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4740.591146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4740.604250] Lustre: Skipped 2 previous similar messages [ 4745.697824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4745.711566] Lustre: Skipped 2 previous similar messages [ 4750.822388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4750.835309] Lustre: Skipped 1 previous similar message [ 4754.400291] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4754.641406] Lustre: server umount lustre-MDT0000 complete [ 4755.942108] LustreError: 152560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4755.967058] LustreError: 152560:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4759.839292] LustreError: 152539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790715297 with bad export cookie 5630058690592867618 [ 4759.856908] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4760.324421] Lustre: server umount lustre-MDT0001 complete [ 4772.500847] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 4775.695901] Key type lgssc unregistered [ 4776.070486] LNet: 156648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4776.080537] LNetError: 156648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4776.103561] LNet: Removed LNI 192.168.203.122@tcp [ 4777.028898] Key type .llcrypt unregistered [ 4777.034119] Key type ._llcrypt unregistered [ 4853.766230] Key type ._llcrypt registered [ 4853.777258] Key type .llcrypt registered [ 4853.943531] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 4855.724665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4855.807410] alg: No test for adler32 (adler32-zlib) [ 4857.142818] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 4857.504696] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 4859.312208] Key type lgssc registered [ 4861.038738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4870.619577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4870.637493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.334251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.726931] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4881.892803] LustreError: 159151:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4881.911698] LustreError: 159151:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4887.020211] LustreError: 159150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4888.625737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4889.297258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4894.744332] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4904.167181] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4910.485651] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4918.495857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4919.051948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4926.226356] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 4934.411406] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4935.219923] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4942.589503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4958.191654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4958.193318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4958.201764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4958.246414] Lustre: Skipped 1 previous similar message [ 4961.987761] Lustre: server umount lustre-OST0000 complete [ 4965.858459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4965.873996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.897540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4965.905333] Lustre: Skipped 1 previous similar message [ 4968.423913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4968.442789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.543878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4973.556784] Lustre: Skipped 2 previous similar messages [ 4978.657301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4978.674017] Lustre: Skipped 1 previous similar message [ 4980.192753] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4980.509966] Lustre: server umount lustre-MDT0000 complete [ 4983.781180] LustreError: 161661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4983.818309] LustreError: 161661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4984.565477] LustreError: 159133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790715522 with bad export cookie 8080700683614396284 [ 4984.572337] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4984.578874] LustreError: 159133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4984.997727] Lustre: server umount lustre-MDT0001 complete [ 4997.957391] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5001.030621] Key type lgssc unregistered [ 5001.360303] LNet: 162714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5001.377633] LNetError: 162714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5001.396921] LNet: Removed LNI 192.168.203.122@tcp [ 5002.451303] Key type .llcrypt unregistered [ 5002.453754] Key type ._llcrypt unregistered [ 5030.901424] Key type ._llcrypt registered [ 5030.903232] Key type .llcrypt registered [ 5031.092579] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5032.978306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5033.259088] alg: No test for adler32 (adler32-zlib) [ 5034.693220] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 5035.170767] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5037.013295] Key type lgssc registered [ 5038.821829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5056.152792] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 16:59:52 (1790715592) [ 5076.952324] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 5088.672538] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5135.618903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5135.660330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5137.027791] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5137.076847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5137.201731] Lustre: lustre-MDT0000: new disk, initializing [ 5137.382310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5137.394280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5142.616644] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5154.805209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.966652] Lustre: 167010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5155.046807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5155.066562] Lustre: Skipped 1 previous similar message [ 5155.149254] Lustre: lustre-MDT0001: new disk, initializing [ 5155.271144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5155.315634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5155.341039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5160.521326] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5171.310069] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5179.365313] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5188.310978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5188.701471] Lustre: lustre-OST0000: new disk, initializing [ 5188.709353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5188.719512] Lustre: 168979:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5188.839586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5197.900341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5197.909233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5198.016889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5198.024609] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5208.895331] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5218.785621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5218.791883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5218.817782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5218.827341] Lustre: Skipped 1 previous similar message [ 5223.926121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5223.934124] Lustre: Skipped 1 previous similar message [ 5224.784906] Lustre: server umount lustre-OST0000 complete [ 5229.026472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.053871] Lustre: Skipped 1 previous similar message [ 5229.057935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5229.075349] Lustre: Skipped 1 previous similar message [ 5234.153655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5234.169150] Lustre: Skipped 1 previous similar message [ 5235.084563] Lustre: server umount lustre-MDT0000 complete [ 5239.654916] LustreError: 167001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790715777 with bad export cookie 12715442282236104787 [ 5239.662751] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5239.671435] LustreError: 167001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5240.138913] Lustre: server umount lustre-MDT0001 complete [ 5252.798567] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 5255.543173] Key type lgssc unregistered [ 5255.833320] LNet: 170882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5255.843556] LNetError: 170882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5255.863719] LNet: Removed LNI 192.168.203.122@tcp [ 5256.673797] Key type .llcrypt unregistered [ 5256.676428] Key type ._llcrypt unregistered [ 5286.649101] Key type ._llcrypt registered [ 5286.657638] Key type .llcrypt registered [ 5286.813240] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 5288.165313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5288.369037] alg: No test for adler32 (adler32-zlib) [ 5289.492384] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 5289.721861] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 5291.408896] Key type lgssc registered [ 5292.666346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5301.183044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5301.206376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5302.594651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.569606] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5311.970926] LustreError: 172316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.988968] LustreError: 172316:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5315.790433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.286586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5321.171347] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5329.265308] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5335.842968] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5343.300428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5343.776357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.906355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5350.279746] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 5358.183939] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5369.306121] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5369.332482] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5369.365988] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5369.390184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.424735] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5370.750327] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5370.775054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.795554] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5370.804517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.819306] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5372.047439] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5372.070384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.091278] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5372.102830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.119275] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5373.091867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.120244] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5374.135161] Lustre: Skipped 1 previous similar message [ 5374.153856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5374.172896] Lustre: Skipped 1 previous similar message [ 5374.182505] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5374.191304] Lustre: Skipped 1 previous similar message [ 5374.200559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.216786] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5374.231521] Lustre: Skipped 1 previous similar message [ 5375.354655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.514508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.838285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.117909] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5379.124047] Lustre: Skipped 3 previous similar messages [ 5379.145512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5379.167225] Lustre: Skipped 3 previous similar messages [ 5379.176735] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5379.191042] Lustre: Skipped 3 previous similar messages [ 5379.194037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.203641] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5379.222951] Lustre: Skipped 3 previous similar messages [ 5380.208453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.391475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.496534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.718158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.808979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.872110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.975764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.090586] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5388.102538] Lustre: Skipped 7 previous similar messages [ 5388.122475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.131270] Lustre: Skipped 7 previous similar messages [ 5388.134842] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5388.139856] Lustre: Skipped 7 previous similar messages [ 5388.144798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.165820] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5388.187523] Lustre: Skipped 7 previous similar messages [ 5389.080471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.319020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.245718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.498727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.583518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.605895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.642583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.614982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.656779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.854589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.763470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.679549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.571818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.719296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.627033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.612397] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5404.622066] Lustre: Skipped 15 previous similar messages [ 5404.637493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5404.645270] Lustre: Skipped 15 previous similar messages [ 5404.655230] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5404.660866] Lustre: Skipped 15 previous similar messages [ 5404.668727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.677775] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5404.685182] Lustre: Skipped 15 previous similar messages [ 5405.568685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.626147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.596129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.657597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.689859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.770930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.883151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.203646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.062236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.005243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.232722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.219081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.306139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.534862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.532904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.487426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.587035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.707278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.093902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.396129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.873886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.249267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.309844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.788214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.071589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.255037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.581641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.731985] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5436.742277] Lustre: Skipped 27 previous similar messages [ 5436.774832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.797189] Lustre: Skipped 27 previous similar messages [ 5436.811180] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5436.822898] Lustre: Skipped 27 previous similar messages [ 5436.827837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.841916] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5436.856074] Lustre: Skipped 27 previous similar messages [ 5437.816715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.909531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.819539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.693240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.720621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.880286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.013868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.028346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.940970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.927464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.222763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.319889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.649445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.968747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.866649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.097815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.516235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.834089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.760593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.039397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.319292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.526609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.596054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.571879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.705360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.860390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.139984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.082403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.193555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.403352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.227048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.369832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.373530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.510713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.536491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.595351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.443384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.440818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.364658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.502785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.483771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.449756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.329917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.283277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.387109] LustreError: 172336:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5485.397423] LustreError: 172336:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5485.411647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.577169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.626809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.726957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.682568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.640123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.555373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.661830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.852958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.806601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.643777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.577606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.690239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.547459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.665637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.677974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.709484] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5501.732336] Lustre: Skipped 60 previous similar messages [ 5501.772232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.821854] Lustre: Skipped 60 previous similar messages [ 5501.835714] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5501.845446] Lustre: Skipped 60 previous similar messages [ 5501.863400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.890031] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5501.903898] Lustre: Skipped 60 previous similar messages [ 5502.802368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.942157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.822739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.844832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.929879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.831541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.979909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.066087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.252855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.459366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.483227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.518661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.648084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.894380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.816969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.100087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.210541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.497239] LustreError: 172336:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5521.520504] LustreError: 172336:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5521.523296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.912653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.292975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.321566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.650096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.775236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.797150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.980695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.002033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.014709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.992382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.146197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.139408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.219191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.297450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.235584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.126467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.221239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.363328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.414099] LustreError: 172336:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5542.429659] LustreError: 172336:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5542.439608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.506200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.532880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.544235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.437618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.403224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.214483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.342443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.210305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.338506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.360100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.515380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.396917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.421221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.388273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.365554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.410592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.486446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.630777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.745929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.685970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.901350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.989160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.446511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.526749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.801461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.902963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.078752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.175197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.516760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.770055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.909706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.957818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.911939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.930904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.938593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.969358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.929942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.787265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.826659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.914888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.166518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.505900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.628460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.684572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.729502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.779855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.908996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.006443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.075719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.029163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.239902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.216489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.215415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.531237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.541466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.473508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.463067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.609765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.442989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.386581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.724741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.760762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.912621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.910732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.960774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.747365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.127921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.508742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.074508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.139276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.436382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.431343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.447382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.229800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.301897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.607843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.421350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.426670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.239682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.482719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.495867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.735718] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5630.741819] Lustre: Skipped 118 previous similar messages [ 5630.756905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.766611] Lustre: Skipped 118 previous similar messages [ 5630.772703] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5630.778619] Lustre: Skipped 118 previous similar messages [ 5630.782509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.794152] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5630.799860] Lustre: Skipped 118 previous similar messages [ 5631.720246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.591287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.541646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.858836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.832559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.058241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.160713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.393614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.447667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.568809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.466411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.268324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.128287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.961626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.957697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.999700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.938736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.946746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.669929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.585553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.451494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.426145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.282485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.140213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.889257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.757453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.596311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.504312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.530272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.532655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.487966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.334806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.107515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.878268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.659450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.584897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.369631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.178191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.114138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.989585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.863057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.856868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.685817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.680814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.690657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.659421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.693456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.582838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.348306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.335962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.225233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.401349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.357802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.457707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.391022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.223688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.142644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.979684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.818504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.806305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.700555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.541528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.526279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.496855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.370641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.351517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.270547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.173400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.083605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.147719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.275917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.095287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.939577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.870417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.129540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.097440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.031440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.873719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.793772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.685424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.684922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.630253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.506595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.531374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.776953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.812357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.739634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.659986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.597328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.673178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.573822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.657578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.593643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.408803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.411150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.413556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.265315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.448642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.471694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.668835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.700332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.706332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.827275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.665712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.804328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.907440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.043760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.092986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.025411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.089690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.986764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.958810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.896391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.965387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.033113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.029735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.986585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.021301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.094301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.934470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.923736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.820928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.104701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.323230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.401614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.307789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.218511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.525072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.422706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.589742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.488922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.461462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.282590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.613064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.833427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.915540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.964649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.147725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.285679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.328618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.421696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.354750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.545411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.527435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.564660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.516835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.637646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.939657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.920966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.982420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.774841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.706633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.661519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.696895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.646360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.797350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.878900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.062156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.278674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.635361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.750692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.190213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.404812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.779067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.483986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.838895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.207080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.609526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.961278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.237334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.278907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.712396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.942505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.868907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.188341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.233211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.670098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.033418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.346226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.479152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.005451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.072450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.114061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.235846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.240554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.447737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.327775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.317705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.251545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.349742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.361752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.400786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.460637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.532623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.648675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.611299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.745908] LustreError: 172336:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5832.765461] LustreError: 172336:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5832.772256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.720605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.804203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.841514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.861067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.866501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.793986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.885596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.716854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.696708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.694341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.722676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.615391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.743414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.813445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.003821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.025582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.951431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.976220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.051504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.999353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.974753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.853364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.953317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.023903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.093863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.997895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.992808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.997254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.384803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.437388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.645717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.512300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.494474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.443703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.316339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.306344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.075056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.093022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.888980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.925511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.727509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.708447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.676843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.671295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.511989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.419438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.461299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.423760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.600262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.719661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.724211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.821701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.715874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.759761] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5886.776568] Lustre: Skipped 250 previous similar messages [ 5886.797459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5886.811730] Lustre: Skipped 250 previous similar messages [ 5886.816340] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5886.822556] Lustre: Skipped 250 previous similar messages [ 5886.826703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.838614] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5886.846482] Lustre: Skipped 250 previous similar messages [ 5887.861491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.933351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.938919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.208853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.203056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.126580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.114486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.941157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.097651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.173376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.244180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.409651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.259127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.249652] LustreError: 172336:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5901.261675] LustreError: 172336:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5901.278678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.123467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.116959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.285190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.405747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.603543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.532807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.472237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.492657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.705047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.973543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.062351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.131950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.197328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.332709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.473698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.977957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.222812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.542923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.909752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.307252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.628198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.811726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.755512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.112991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.141246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.389876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.777114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.903087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.157490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.514209] LustreError: 172336:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5936.533538] LustreError: 172336:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5936.555878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.621679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.169852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.650264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.963560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.575263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.690356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.665422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.707428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.703105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.779729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.861791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.935373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.783345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.758499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.763197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.822392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.732140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.753364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.777356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.846451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.140646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.459541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.563667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.708997] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.807806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.104662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.248983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.233973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.169783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.202040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.629992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.878123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.093302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.449566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.648218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.776635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.785319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.916248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.969742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.113081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.072150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.109844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.051554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.095646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.988943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.106127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.238653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.166144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.274184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.333182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.351606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.185196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.212768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.083961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.157711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.165579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.153361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.057531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.997661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.001038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.978040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.187670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.302836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.540354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.864732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.980155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.117634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.514523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.124163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.940746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.046730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.351072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.508199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.750602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.927761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.219407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.619472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.017747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.643346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.827305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.994164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.074558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.196808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.212641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.365062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.321288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.529074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.485318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.572319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.694266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.570104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.471145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.474484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.523804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.500217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.454661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.253790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.099853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.953898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.009495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.268147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.453412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.396064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.512325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.713857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.590477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.615461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.801567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.783155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.011168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.172892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.272711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.409498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.609808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.598805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.812851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.682835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.741474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.506617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.468059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.274847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.091120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.923882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.973519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.845884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.956164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.963530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.102666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.010475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.011977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.806571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.797315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.859121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.973082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.158686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.112118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.036684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.980515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.302161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.508081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.769192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.912147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.882060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.986332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.072655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.128375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.019796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.040108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.080809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.271185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.274188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.241980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.154370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.156349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.175220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.303047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.416536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.665986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.987481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.842147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.865261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.867450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.831619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.819678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.159067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.373083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.372705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.465547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.358269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.487474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.569159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.521678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.671900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.652100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.738685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.900752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.895164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.078708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.263600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.230214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.236655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.427970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.410245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.681758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.899053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.061537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.013283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.016806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.995191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.231388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.250206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.385414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.573764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.635875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.877534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.914030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.092117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.141182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.318607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.391787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.373509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.502886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.557919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.506493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.730741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.660850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.704027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.651772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.828730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.795671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.963194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.955565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.019162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.195068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.482762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.614469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.847542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.169680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.268059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.281436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.227338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.289909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.450486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.280308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.298520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.555630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.711397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.914044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.024641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.176723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.320368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.391399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.268598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.370044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.333894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.197191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.115784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.863517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.904126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.060953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.015302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.939608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.823972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.818145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.899912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.790622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.727375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.572904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.571804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.482914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.521587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.417996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.378907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.349845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.316474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.248613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.145231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.996154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.840960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.799476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.686501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.564639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.589435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.557973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.616762] LustreError: 172336:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6220.631952] LustreError: 172336:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6220.657234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.735439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.757623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.715954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.675144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.533603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.309467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.192945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.008765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.064194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.192026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.229206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.390475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.506802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.396408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.334875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.233679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.235125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.042877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.122468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.147098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.145581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.079055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.015429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.780435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.812640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.784937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.800772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.759654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.783270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.751610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.695228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.643786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.928887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.028654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.019523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.902899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.668259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.616646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.615900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.482977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.277729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.201729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.025791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.157317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.080704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.970960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.888591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.927309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.911947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.023186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.975049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.974340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.928534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.831283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.755067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.771061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.665485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.655750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.581565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.751564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.783049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.602710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.389045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.188154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.996366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.755595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.602554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.518907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.674848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.604553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.566525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.414243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.505630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.570252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.622631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.568256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.839831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.687576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.806409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.097203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.146906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.142078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.331473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.384618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.436398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.554353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.794790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.642490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.866442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.851962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.288791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.233287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.218278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.415189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.341880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.219187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.385291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.274480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.228165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.335694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.377247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.495105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.525749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.447710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.742529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.706449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.750662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.767389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.904769] LustreError: 172336:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6328.919164] LustreError: 172336:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6328.922154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.056654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.071680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.937636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.995721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.944767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.040223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.011368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.048858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.004754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.861234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.928454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.981951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.101713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.116176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.127411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.103194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.172083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.145483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.340212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.348926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.639820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.654361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.832196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.784919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.834030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.848257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.956197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.920610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.913504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.808684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.017830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.200060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.244646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.359256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.735347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.988876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.450679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.541834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.411222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.364909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.295201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.379938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.292185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.345547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.203854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.272705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.260742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.389377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.675966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.865299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.212417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.432781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.592159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.883183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.098123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.489682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.598549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.726806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.765192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.910447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.022798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.924676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.022832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.056247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.108329] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6399.112693] Lustre: Skipped 482 previous similar messages [ 6399.125700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6399.133370] Lustre: Skipped 482 previous similar messages [ 6399.136942] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6399.145726] Lustre: Skipped 482 previous similar messages [ 6399.157918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.182053] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6399.207196] Lustre: Skipped 482 previous similar messages [ 6400.315695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.448613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.584485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.868636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.990193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.120091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.198168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.219116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.425962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.429587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.718896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.006911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.291284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.438426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.652788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.040737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.240991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.339126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.511681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.660266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.797302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.917619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.017336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.936342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.918239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6883.795442] LustreError: 174362:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6883.807263] LustreError: 172336:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7603.681891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7603.688660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7603.702304] Lustre: Skipped 25 previous similar messages [ 7603.709208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7604.729174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7604.744680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7604.745155] LustreError: Skipped 4 previous similar messages [ 7605.198426] Lustre: server umount lustre-OST0000 complete [ 7609.829625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7609.836913] LustreError: Skipped 3 previous similar messages [ 7609.846955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7612.897944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7612.904318] Lustre: Skipped 1 previous similar message [ 7615.595905] Lustre: server umount lustre-MDT0000 complete [ 7618.018629] LustreError: 173558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7618.042925] LustreError: 173558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7619.426205] LustreError: 208690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718157 with bad export cookie 6092136769740297761 [ 7619.444914] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7619.451674] LustreError: 208690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7619.842800] Lustre: server umount lustre-MDT0001 complete [ 7632.137411] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7635.151956] Key type lgssc unregistered [ 7635.460343] LNet: 227608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7635.467373] LNetError: 227608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7635.495166] LNet: Removed LNI 192.168.203.122@tcp [ 7636.497747] Key type .llcrypt unregistered [ 7636.499663] Key type ._llcrypt unregistered [ 7663.990488] Key type ._llcrypt registered [ 7663.992712] Key type .llcrypt registered [ 7664.097149] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7665.539096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7665.569987] alg: No test for adler32 (adler32-zlib) [ 7666.794256] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 7667.076776] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7668.752341] Key type lgssc registered [ 7670.120202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7677.900102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7677.926247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7679.556244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7684.599376] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7689.707936] LustreError: 229036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7689.728068] LustreError: 229036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7694.439679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.814727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7699.611226] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7707.030365] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7714.024294] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7720.873954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.276667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.363036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7727.698537] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7735.565161] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7743.632338] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7745.319513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.336506] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7745.342994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7745.352469] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7745.356300] LustreError: 229056:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7750.118278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7750.118466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7750.141804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7750.154372] Lustre: Skipped 1 previous similar message [ 7752.659398] Lustre: lustre-OST0000: Not available for connect from 192.168.203.22@tcp (stopping) [ 7754.001289] Lustre: server umount lustre-OST0000 complete [ 7765.417088] LustreError: 229056:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7788.765537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7789.230502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.254482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7790.272516] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7790.280027] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7795.731200] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7803.927253] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7805.848287] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7812.253138] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88d788364800.ost_server_uuid 50 [ 7813.980519] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88d788364800.ost_server_uuid in FULL state after 0 sec [ 7819.240925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7819.249098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.270900] Lustre: Skipped 1 previous similar message [ 7819.281618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.281524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7821.289041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.294027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7824.691396] Lustre: server umount lustre-OST0000 complete [ 7831.550249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7831.557880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7831.577477] Lustre: Skipped 1 previous similar message [ 7831.602133] Lustre: Skipped 2 previous similar messages [ 7834.612842] LustreError: 233495:0:(ldlm_resource.c:1227:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000777cc7d4) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7834.870322] Lustre: server umount lustre-MDT0000 complete [ 7836.644105] LustreError: 229037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7836.678333] LustreError: 229037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7838.754991] LustreError: 229019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718376 with bad export cookie 18101749110424424245 [ 7838.758524] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7838.767184] LustreError: 229019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7839.142433] Lustre: server umount lustre-MDT0001 complete [ 7851.006892] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 7853.947982] Key type lgssc unregistered [ 7854.294261] LNet: 234245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7854.315080] LNetError: 234245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7854.341963] LNet: Removed LNI 192.168.203.122@tcp [ 7855.342144] Key type .llcrypt unregistered [ 7855.345683] Key type ._llcrypt unregistered [ 7887.809941] Key type ._llcrypt registered [ 7887.815941] Key type .llcrypt registered [ 7887.925365] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 7889.022874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7889.060172] alg: No test for adler32 (adler32-zlib) [ 7890.136575] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 7890.341141] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 7892.000357] Key type lgssc registered [ 7893.190626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7901.675921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7901.720310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7903.152816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7907.148851] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7913.443345] LustreError: 235989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.459807] LustreError: 235989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7915.659711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.053238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7920.452518] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7922.961662] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7928.484566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.814677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7934.282116] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7943.076565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7943.311577] Lustre: lustre-OST0001: new disk, initializing [ 7943.316409] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7943.327367] Lustre: 238296:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7943.450464] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7948.610980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7948.911368] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 7950.437096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7950.449868] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7950.549419] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7959.317393] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7968.142916] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7980.845295] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7980.857754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.869937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7981.539433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7981.552143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.565487] Lustre: Skipped 2 previous similar messages [ 7986.663576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7986.672656] Lustre: Skipped 5 previous similar messages [ 7991.780696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7991.788060] Lustre: Skipped 3 previous similar messages [ 7995.360605] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7995.603917] Lustre: server umount lustre-MDT0000 complete [ 7996.903161] LustreError: 235989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7996.927757] LustreError: 235989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7998.658507] LustreError: 239067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718536 with bad export cookie 8213649151920764421 [ 7998.659976] LustreError: 236727:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200001b72:0x1:0x0]: rc = -5 [ 7998.661276] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7998.674024] LustreError: 239067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7999.002462] Lustre: server umount lustre-MDT0001 complete [ 8012.756027] Lustre: server umount lustre-OST0000 complete [ 8025.385308] Lustre: server umount lustre-OST0001 complete [ 8048.425530] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8059.294828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8059.777353] LustreError: 241403:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8059.837808] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8064.121721] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8065.003496] LustreError: 241404:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.091366] LustreError: 241403:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8071.878536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.332516] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8076.778830] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8079.721916] Lustre: 242545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8086.447794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8092.873845] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8097.070626] LustreError: 242897:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8097.084530] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 8101.018375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8101.287117] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8101.296246] Lustre: Skipped 1 previous similar message [ 8104.379693] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8107.517357] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8115.538701] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8118.872587] Lustre: 244416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8130.023684] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8130.033586] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8130.047163] Lustre: Skipped 2 previous similar messages [ 8130.062743] Lustre: Skipped 3 previous similar messages [ 8135.616931] Lustre: server umount mylustre-MDT0000 complete [ 8139.562363] LustreError: 241380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718677 with bad export cookie 8213649151920767284 [ 8139.573839] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8139.575963] LustreError: 241380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8139.861500] Lustre: server umount mylustre-MDT0001 complete [ 8154.072421] Lustre: server umount mylustre-OST0000 complete [ 8155.937718] Lustre: 235245:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790718678/real 1790718678] req@ffff9a9d3db4c700 x1877704366515840/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790718694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8155.991904] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8156.017471] Lustre: Skipped 1 previous similar message [ 8157.817497] Lustre: server umount mylustre-OST0001 complete [ 8178.837555] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8188.823542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.226363] LustreError: 246727:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8189.286576] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8193.561685] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8201.972393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8206.243918] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8208.978587] Lustre: 247870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8215.296615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.622968] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8226.915292] LustreError: 248225:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.927725] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:161) [ 8226.951610] LustreError: 248225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8229.943484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8234.237596] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8234.240111] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8235.925885] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8243.519085] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8247.002607] Lustre: 249742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8254.955532] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.965094] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.967149] Lustre: Skipped 1 previous similar message [ 8254.977748] Lustre: Skipped 7 previous similar messages [ 8259.991974] Lustre: server umount tfs-MDT0000 complete [ 8263.692913] LustreError: 246704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718801 with bad export cookie 8213649151920769237 [ 8263.694210] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8263.701749] LustreError: 246704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8264.038346] Lustre: server umount tfs-MDT0001 complete [ 8277.124445] Lustre: server umount tfs-OST0000 complete [ 8291.498355] Lustre: server umount tfs-OST0001 complete [ 8315.240861] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8325.711062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8326.116369] LustreError: 251994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8326.150765] LustreError: 251994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8326.239795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8326.248512] Lustre: Skipped 3 previous similar messages [ 8331.388384] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8340.319622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8345.417631] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8348.448302] Lustre: 253136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8355.732707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8363.399217] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8365.446189] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:193) [ 8371.727985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8372.563863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8377.862392] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8378.774206] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8386.451261] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8389.815807] Lustre: 255006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8396.927527] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 17:55:33 (1790718933) [ 8413.671021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.675165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8413.692849] Lustre: Skipped 5 previous similar messages [ 8413.709409] Lustre: Skipped 3 previous similar messages [ 8419.857105] Lustre: server umount lustre-MDT0000 complete [ 8424.049434] LustreError: 251970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790718962 with bad export cookie 8213649151920771155 [ 8424.063407] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8424.067643] LustreError: 251970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8430.672355] Lustre: server umount lustre-MDT0001 complete [ 8440.781298] Lustre: server umount lustre-OST0000 complete [ 8450.922918] Lustre: server umount lustre-OST0001 complete [ 8457.553502] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8468.530303] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8503.730460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8503.922482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8503.941906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8504.010508] Lustre: lustre-MDT0000: new disk, initializing [ 8504.067685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8504.073530] Lustre: Skipped 3 previous similar messages [ 8504.088701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8508.777554] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8520.963960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.120151] Lustre: 258191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8521.210488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8521.216946] Lustre: Skipped 1 previous similar message [ 8521.327785] Lustre: lustre-MDT0001: new disk, initializing [ 8521.520226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8521.534765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8526.467342] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8536.680455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8536.978990] Lustre: lustre-OST0000: new disk, initializing [ 8536.988788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8537.002183] Lustre: 259773:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8538.916527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8538.925487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8539.069779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8543.137949] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8553.768546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8553.924712] Lustre: lustre-OST0001: new disk, initializing [ 8553.928512] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8553.937935] Lustre: 260644:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8555.205771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8555.222943] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8555.325293] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8561.293481] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8573.509591] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 17:58:30 (1790719110) [ 8588.496883] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 17:58:45 (1790719125) [ 8605.670021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8605.674825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8605.681651] LustreError: Skipped 1 previous similar message [ 8605.686859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8605.696020] Lustre: Skipped 3 previous similar messages [ 8605.726510] Lustre: Skipped 7 previous similar messages [ 8609.986412] Lustre: server umount lustre-OST0000 complete [ 8611.814923] LustreError: 261568:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8611.840667] LustreError: 261568:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 8620.559966] Lustre: server umount lustre-MDT0000 complete [ 8624.258976] LustreError: 258182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719162 with bad export cookie 8213649151920772933 [ 8624.261983] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8624.268538] LustreError: 258182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8630.835678] Lustre: server umount lustre-MDT0001 complete [ 8644.524870] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8718.387754] Lustre: server umount lustre-OST0001 complete [ 8725.151102] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 8740.836584] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8778.493949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8778.741588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8778.774187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8778.829342] Lustre: lustre-MDT0000: new disk, initializing [ 8778.879202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8778.885763] Lustre: Skipped 3 previous similar messages [ 8778.896885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8783.429482] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8793.581077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8793.741156] Lustre: 265509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8794.118665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8798.984672] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8808.911902] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8814.312120] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8820.856969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8821.293742] Lustre: lustre-OST0000: new disk, initializing [ 8821.296449] Lustre: Skipped 1 previous similar message [ 8821.301838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8821.306518] Lustre: Skipped 2 previous similar messages [ 8821.311319] Lustre: 267479:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8822.601669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8822.615090] Lustre: Skipped 1 previous similar message [ 8822.617800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8822.731087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8826.595469] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8835.323945] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8843.237439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8843.237525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.265168] LustreError: Skipped 1 previous similar message [ 8843.268922] Lustre: Skipped 4 previous similar messages [ 8843.279959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8843.294231] Lustre: Skipped 8 previous similar messages [ 8848.605111] Lustre: server umount lustre-OST0000 complete [ 8858.960748] Lustre: server umount lustre-MDT0000 complete [ 8863.497102] LustreError: 267478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719401 with bad export cookie 8213649151920774403 [ 8863.505976] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8863.506615] LustreError: 267478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8863.951753] Lustre: server umount lustre-MDT0001 complete [ 8878.429637] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 8881.908051] Key type lgssc unregistered [ 8882.315911] LNet: 269240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8882.327885] LNetError: 269240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8882.344048] LNet: Removed LNI 192.168.203.122@tcp [ 8883.517470] Key type .llcrypt unregistered [ 8883.520783] Key type ._llcrypt unregistered [ 8915.641674] Key type ._llcrypt registered [ 8915.643499] Key type .llcrypt registered [ 8915.801625] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 8917.219943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8917.339291] alg: No test for adler32 (adler32-zlib) [ 8918.598081] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 8918.955731] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 8920.688197] Key type lgssc registered [ 8922.400786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8931.894042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 8931.919704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.367842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8937.465684] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8943.590627] LustreError: 270720:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.601289] LustreError: 270720:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8946.240631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.551959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8951.439198] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8960.211425] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8968.284728] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8975.533634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8975.953340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8978.043636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8982.902656] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 8992.867783] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8994.098905] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8996.842860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8996.850348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8996.852947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8996.893480] Lustre: Skipped 1 previous similar message [ 8998.707965] Lustre: server umount lustre-OST0000 complete [ 9007.079469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9007.082245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9007.089070] Lustre: Skipped 1 previous similar message [ 9007.093404] Lustre: Skipped 1 previous similar message [ 9012.194077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9012.197581] Lustre: Skipped 2 previous similar messages [ 9017.314972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.324320] Lustre: Skipped 1 previous similar message [ 9018.336178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9018.574162] Lustre: server umount lustre-MDT0000 complete [ 9022.434301] LustreError: 270717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9022.443569] LustreError: 270717:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9023.345545] LustreError: 270700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719561 with bad export cookie 8662195357670724113 [ 9023.370140] LustreError: 270700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9023.377510] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9023.952675] Lustre: server umount lustre-MDT0001 complete [ 9036.695490] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9039.892244] Key type lgssc unregistered [ 9040.205174] LNet: 274304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9040.222776] LNetError: 274304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9040.238066] LNet: Removed LNI 192.168.203.122@tcp [ 9041.139971] Key type .llcrypt unregistered [ 9041.142694] Key type ._llcrypt unregistered [ 9058.100669] Key type ._llcrypt registered [ 9058.103589] Key type .llcrypt registered [ 9058.211477] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9059.132781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9059.266214] alg: No test for adler32 (adler32-zlib) [ 9060.325634] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 9060.555820] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9062.248728] Key type lgssc registered [ 9063.358806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9072.276194] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9072.321606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9087.904637] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9093.024310] LustreError: 275513:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.122@tcp: failed processing log, type 4: rc = -110 [ 9119.714751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9125.768763] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9133.913269] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9136.558841] Lustre: server umount lustre-OST0000 complete [ 9154.430614] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9157.142858] Key type lgssc unregistered [ 9157.416477] LNet: 277142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9157.429769] LNetError: 277142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9158.506977] LNet: Removed LNI 192.168.203.122@tcp [ 9159.417982] Key type .llcrypt unregistered [ 9159.419810] Key type ._llcrypt unregistered [ 9197.996978] Key type ._llcrypt registered [ 9197.998762] Key type .llcrypt registered [ 9198.126446] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9199.247246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9199.376701] alg: No test for adler32 (adler32-zlib) [ 9200.557352] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 9200.787488] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9202.442346] Key type lgssc registered [ 9204.101448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9218.409087] loop0: detected capacity change from 0 to 204800000 [ 9218.705364] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9218.725527] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9219.136818] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9220.020049] loop0: detected capacity change from 0 to 204800000 [ 9220.181765] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9220.195099] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9220.605411] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9221.626423] loop0: detected capacity change from 0 to 204800000 [ 9221.763378] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9221.780285] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9222.021837] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9223.056520] loop0: detected capacity change from 0 to 204800000 [ 9223.264654] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9224.420526] loop0: detected capacity change from 0 to 204800000 [ 9224.440525] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9229.132436] loop0: detected capacity change from 0 to 204800000 [ 9229.186459] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9234.364113] loop0: detected capacity change from 0 to 204800000 [ 9234.406090] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9239.920767] loop0: detected capacity change from 0 to 204800000 [ 9239.955490] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9245.204896] loop0: detected capacity change from 0 to 204800000 [ 9245.253129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9245.284676] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.339474] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9245.350762] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9245.361902] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9245.370789] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9245.385456] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9245.396471] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9245.425757] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9245.455641] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9245.492627] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9245.526451] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9245.549816] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9245.580081] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9245.608771] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9245.627902] Lustre: 280572:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9252.719835] Lustre: server umount MGS complete [ 9253.517673] loop0: detected capacity change from 0 to 204800000 [ 9253.643325] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9254.411207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9254.492413] Lustre: lustre-MDT0000: Aborting client recovery [ 9255.650104] loop1: detected capacity change from 0 to 204800000 [ 9255.798805] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.983858] Lustre: 281006:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9256.017056] Lustre: 281006:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9256.054749] Lustre: 281006:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9256.075986] Lustre: 281006:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9256.103704] Lustre: 281006:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9256.122558] Lustre: 281006:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9256.140566] Lustre: 281006:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9256.187708] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9256.415816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9256.498752] Lustre: lustre-MDT0001: Aborting client recovery [ 9257.585756] loop2: detected capacity change from 0 to 204800000 [ 9257.738983] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9257.871172] Lustre: 281084:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9257.887955] Lustre: 281084:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9257.928320] Lustre: 281084:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9257.939331] Lustre: 281084:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9257.946448] Lustre: 281084:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9257.958827] Lustre: 281084:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9257.967742] Lustre: 281084:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9258.102153] Lustre: Found index 0 for lustre-OST0000, updating log [ 9258.142556] Lustre: 281102:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9258.227660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9258.253787] Lustre: lustre-OST0000: Aborting recovery [ 9259.272230] loop3: detected capacity change from 0 to 204800000 [ 9259.466886] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9259.534277] Lustre: 281167:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9259.549375] Lustre: 281167:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9259.571398] Lustre: 281167:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9259.585829] Lustre: 281167:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9259.604644] Lustre: 281167:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9259.633455] Lustre: 281167:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9259.655599] Lustre: 281167:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9259.726138] Lustre: Found index 1 for lustre-OST0001, updating log [ 9259.822116] Lustre: 281174:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9259.903560] Lustre: lustre-OST0001: Aborting recovery [ 9263.999218] LustreError: 280950:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9264.000189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9264.011761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9264.031780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9264.064683] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9266.032692] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9266.032726] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9269.294703] LustreError: 280917:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1069us: rc = -2 [ 9269.801116] LustreError: 280916:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1059us: rc = -2 [ 9269.830866] LustreError: 280916:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 15 previous similar messages [ 9286.628463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9286.631493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9286.659716] Lustre: Skipped 2 previous similar messages [ 9286.673189] Lustre: Skipped 3 previous similar messages [ 9291.748723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9291.761100] Lustre: Skipped 3 previous similar messages [ 9296.878340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9296.885741] Lustre: Skipped 3 previous similar messages [ 9297.377139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9297.847303] Lustre: server umount lustre-MDT0000 complete [ 9299.014362] LustreError: 280901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790719837 with bad export cookie 2310182891746723613 [ 9299.015915] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9299.027865] LustreError: 280901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9301.995475] LustreError: 280915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9301.998422] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9302.026332] Lustre: Skipped 1 previous similar message [ 9302.026903] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9302.066340] LustreError: 280915:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9304.995169] LustreError: 280915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9305.022945] LustreError: 280915:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9305.759050] Lustre: server umount lustre-MDT0001 complete [ 9313.098908] Lustre: server umount lustre-OST0000 complete [ 9320.480301] Lustre: server umount lustre-OST0001 complete [ 9329.555875] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 18:11:06 (1790719866) [ 9353.256141] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9394.603027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9394.843204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9394.887314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9394.976887] Lustre: lustre-MDT0000: new disk, initializing [ 9395.083203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9395.090823] Lustre: Skipped 1 previous similar message [ 9395.133920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9399.718359] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9406.537720] Lustre: server umount lustre-MDT0000 complete [ 9425.799265] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 18:12:42 (1790719962) [ 9455.782849] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 9465.810576] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9504.858779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9505.045180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9505.073510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9505.086253] Lustre: Skipped 1 previous similar message [ 9505.151637] Lustre: lustre-MDT0000: new disk, initializing [ 9505.275314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9505.309237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9509.736984] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9521.301976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9521.448560] Lustre: 288278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9521.534262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9521.544374] Lustre: Skipped 1 previous similar message [ 9521.687449] Lustre: lustre-MDT0001: new disk, initializing [ 9521.813649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9521.871207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9521.896783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9526.178033] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9536.274542] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9542.302649] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9550.794367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9551.171422] Lustre: lustre-OST0000: new disk, initializing [ 9551.182607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9551.195265] Lustre: 290248:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9551.399563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9557.588405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9557.607972] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9557.724156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9558.086653] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9569.551401] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9575.445830] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9590.069329] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9596.158206] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9645.031249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9645.056715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9645.064255] Lustre: Skipped 2 previous similar messages [ 9645.067456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9645.110902] Lustre: Skipped 4 previous similar messages [ 9655.267928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9655.284137] Lustre: Skipped 3 previous similar messages [ 9658.338705] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9658.564154] Lustre: server umount lustre-OST0000 complete [ 9665.514858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9665.514985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9665.545427] Lustre: Skipped 1 previous similar message [ 9669.900995] Lustre: server umount lustre-MDT0000 complete [ 9670.625526] LustreError: 288291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.652412] LustreError: 288291:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9674.229795] LustreError: 288272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720212 with bad export cookie 2310182891746726595 [ 9674.231835] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9674.240809] LustreError: 288272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9674.633602] Lustre: server umount lustre-MDT0001 complete [ 9684.287231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9689.797599] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9695.431984] Lustre: server umount MGS complete [ 9707.999780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9708.500026] LustreError: 294172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9708.547751] LustreError: 294185:0:(qmt_pool.c:1648:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9708.623890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9713.640614] LustreError: 294173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9713.777563] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9723.699978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9728.577663] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9737.512820] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9745.704458] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9753.867815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9760.428076] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9768.776428] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9809.378536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9809.382907] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.388309] LustreError: Skipped 1 previous similar message [ 9809.395579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9809.407068] Lustre: Skipped 1 previous similar message [ 9809.435074] Lustre: Skipped 3 previous similar messages [ 9814.460450] Lustre: server umount lustre-OST0000 complete [ 9819.618203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9819.628834] Lustre: Skipped 1 previous similar message [ 9824.917490] Lustre: server umount lustre-MDT0000 complete [ 9829.080977] LustreError: 294155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720367 with bad export cookie 2310182891746728520 [ 9829.087891] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9829.097414] LustreError: 294155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9829.535562] Lustre: server umount lustre-MDT0001 complete [ 9844.585036] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [ 9848.893500] Key type lgssc unregistered [ 9849.217144] LNet: 298139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9849.250138] LNetError: 298139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9849.291974] LNet: Removed LNI 192.168.203.122@tcp [ 9850.727172] Key type .llcrypt unregistered [ 9850.728746] Key type ._llcrypt unregistered [ 9893.829536] Key type ._llcrypt registered [ 9893.835458] Key type .llcrypt registered [ 9893.967667] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [ 9911.519247] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [ 9912.520771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9912.598374] alg: No test for adler32 (adler32-zlib) [ 9913.709799] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [ 9913.929402] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [ 9915.608494] Key type lgssc registered [ 9916.990917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9958.313102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 9958.402759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.612948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9959.654497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9959.737162] Lustre: lustre-MDT0000: new disk, initializing [ 9959.892429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9959.911348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9964.899726] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9976.130324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.352871] Lustre: 302116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9976.456208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9976.460852] Lustre: Skipped 1 previous similar message [ 9976.625573] Lustre: lustre-MDT0001: new disk, initializing [ 9976.817516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9976.880191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9976.895474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9982.027925] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [ 9992.537626] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9998.277394] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10006.506362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10006.843620] Lustre: lustre-OST0000: new disk, initializing [10006.865284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10006.875893] Lustre: 304083:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10006.995915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10013.318259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10013.336947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10013.475421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10014.189534] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10025.378089] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10032.372939] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10046.554267] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10052.715250] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10110.949195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10110.953440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.961307] LustreError: Skipped 1 previous similar message [10110.970019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.985176] Lustre: Skipped 1 previous similar message [10116.075613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10116.084939] Lustre: Skipped 1 previous similar message [10121.204928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10121.212212] Lustre: Skipped 1 previous similar message [10122.208968] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10122.507446] Lustre: server umount lustre-OST0000 complete [10130.407491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10130.420399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10130.442341] Lustre: Skipped 1 previous similar message [10130.450141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10131.427719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10132.767664] Lustre: server umount lustre-MDT0000 complete [10137.108116] LustreError: 302109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720675 with bad export cookie 9098008364858137158 [10137.120321] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10137.123346] LustreError: 302109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10137.560598] Lustre: server umount lustre-MDT0001 complete [10146.379781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10151.507326] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10157.335378] Lustre: server umount MGS complete [10170.077133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10170.535914] LustreError: 308026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10170.568112] LustreError: 308026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10170.574816] LustreError: 308039:0:(qmt_pool.c:1648:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10170.665148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10175.705081] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10175.974686] LustreError: 308027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10180.070230] LustreError: 308026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10184.886104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10185.290425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10190.734246] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10199.717631] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10205.691363] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10213.126253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10213.546817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10220.361678] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10228.263894] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10260.964615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.974623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10260.989538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10260.989547] Lustre: Skipped 2 previous similar messages [10263.414157] Lustre: server umount lustre-OST0000 complete [10271.206804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10271.213704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10271.236034] Lustre: Skipped 2 previous similar messages [10271.263573] Lustre: Skipped 2 previous similar messages [10275.374188] Lustre: server umount lustre-MDT0000 complete [10276.325409] LustreError: 308021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.354387] LustreError: 308021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10281.268310] LustreError: 308006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790720819 with bad export cookie 9098008364858138999 [10281.277168] LustreError: 308006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10281.278438] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10281.826169] Lustre: server umount lustre-MDT0001 complete [10293.376672] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [10296.319890] Key type lgssc unregistered [10296.717659] LNet: 311971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10296.738313] LNetError: 311971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10296.771308] LNet: Removed LNI 192.168.203.122@tcp [10297.763717] Key type .llcrypt unregistered [10297.768351] Key type ._llcrypt unregistered [10322.370986] Key type ._llcrypt registered [10322.373454] Key type .llcrypt registered [10322.509276] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [10340.648285] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [10341.831353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10341.911667] alg: No test for adler32 (adler32-zlib) [10343.019579] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [10343.345723] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [10345.056693] Key type lgssc registered [10346.286540] Lustre: Echo OBD driver; http://www.lustre.org/ [10369.813559] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [10384.056858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [10384.088466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10385.273209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10385.305333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10385.371635] Lustre: lustre-MDT0000: new disk, initializing [10385.445163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10385.463929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10389.598516] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [10399.335325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10399.527414] Lustre: lustre-OST0000: new disk, initializing [10399.530706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10399.534210] Lustre: Skipped 1 previous similar message [10399.538166] Lustre: 316036:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10399.602188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10404.960577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10404.978334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10405.053063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10406.569553] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11344.870659] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.898625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11345.436543] Lustre: server umount lustre-MDT0000 complete [11349.290408] LustreError: 315223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790721887 with bad export cookie 3489220683035851081 [11349.298815] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11349.301316] LustreError: 315223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11349.506241] Lustre: server umount lustre-OST0000 complete [11374.227657] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [11384.129418] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11421.355732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11421.591423] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11421.620716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11421.681990] Lustre: lustre-MDT0000: new disk, initializing [11421.742227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11421.768714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11426.665882] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11437.487974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11437.622521] Lustre: 320397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11437.698546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11437.702192] Lustre: Skipped 1 previous similar message [11437.804423] Lustre: lustre-MDT0001: new disk, initializing [11437.906314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11437.952304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11437.962750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11442.663575] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11451.781803] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11457.535401] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11464.291384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11464.524521] Lustre: lustre-OST0000: new disk, initializing [11464.527968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11464.532866] Lustre: 322366:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11464.592436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11469.900063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11469.914797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11470.070616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11470.931190] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11480.238506] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11485.145725] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11485.394237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11490.276816] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11490.516389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11495.913690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11495.919661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11495.925437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11495.945033] Lustre: Skipped 1 previous similar message [11495.979479] LustreError: Skipped 1 previous similar message [11501.036166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11501.043347] Lustre: Skipped 2 previous similar messages [11506.159462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11506.166589] Lustre: Skipped 1 previous similar message [11507.168312] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11507.342376] Lustre: server umount lustre-OST0000 complete [11514.850844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11514.861838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11514.878519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11517.720848] Lustre: server umount lustre-MDT0000 complete [11521.958694] LustreError: 320388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722060 with bad export cookie 3489220683038791900 [11521.963619] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11521.974348] LustreError: 320388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11522.305843] Lustre: server umount lustre-MDT0001 complete [11525.715360] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11527.774169] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 18:47:44 (1790722064) [11534.811019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11535.234277] LustreError: 324681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11535.247302] LustreError: 324681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11535.332473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11539.601032] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11540.461602] LustreError: 324682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11545.576804] LustreError: 324681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11547.872997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11548.224591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11552.377881] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11559.490835] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11564.949158] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11571.573349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11572.106493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11579.194827] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11587.245133] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11594.750465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11595.053779] Lustre: lustre-OST0001: new disk, initializing [11595.061684] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11595.078760] Lustre: 327515:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11595.218343] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11601.525361] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11605.131262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11605.141828] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11605.226989] LustreError: 328007:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11606.347125] LustreError: 328013:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11606.358865] LustreError: 328013:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11607.396913] LustreError: 328007:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11607.422168] LustreError: 328007:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11609.451782] LustreError: 328013:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11609.468746] LustreError: 328013:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11611.842574] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11612.052918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11613.608296] LustreError: 328013:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11613.624817] LustreError: 328013:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [11617.557951] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88d7840a0800.ost_server_uuid 50 [11619.031454] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88d7840a0800.ost_server_uuid in FULL state after 0 sec [11621.806531] LustreError: 328007:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11621.816694] LustreError: 328007:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [11624.011737] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88d7840a0800.ost_server_uuid 50 [11625.894664] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88d7840a0800.ost_server_uuid in FULL state after 0 sec [11628.146097] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11641.315038] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11641.332114] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11641.335951] Lustre: Skipped 2 previous similar messages [11641.352155] Lustre: Skipped 3 previous similar messages [11655.136168] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11655.407893] Lustre: server umount lustre-OST0001 complete [11656.684176] LustreError: 327805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11656.710579] LustreError: 327805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11661.799988] LustreError: 327805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11661.815830] LustreError: 327805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11662.305795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11662.313311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11662.345623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11662.353100] Lustre: Skipped 7 previous similar messages [11665.384861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11668.173943] Lustre: server umount lustre-OST0000 complete [11672.034586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11672.050686] Lustre: Skipped 1 previous similar message [11686.368151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11686.651505] Lustre: server umount lustre-MDT0000 complete [11687.393853] LustreError: 324677:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11687.413500] LustreError: 324677:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11690.838099] LustreError: 324661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722228 with bad export cookie 3489220683038792782 [11690.845610] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11690.851982] LustreError: 324661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11691.224610] Lustre: server umount lustre-MDT0001 complete [11701.753445] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11704.567797] Key type lgssc unregistered [11704.873649] LNet: 330206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11704.879380] LNetError: 330206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11704.893547] LNet: Removed LNI 192.168.203.122@tcp [11705.693806] Key type .llcrypt unregistered [11705.695195] Key type ._llcrypt unregistered [11728.583731] Key type ._llcrypt registered [11728.585501] Key type .llcrypt registered [11728.699687] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [11729.761072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11729.775531] alg: No test for adler32 (adler32-zlib) [11730.803207] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [11731.013054] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [11732.712373] Key type lgssc registered [11733.898707] Lustre: Echo OBD driver; http://www.lustre.org/ [11741.123350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [11741.157464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.618066] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11747.958758] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11751.974992] LustreError: 331642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11751.992205] LustreError: 331642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11756.307890] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11757.029373] LustreError: 331643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.153385] LustreError: 331642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11763.854246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11764.304859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11768.796244] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11778.804700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11779.163138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11785.537181] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11787.376686] LustreError: 333322:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11787.413144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11794.363136] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11794.923564] LustreError: 333323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11794.932795] LustreError: 333323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11801.499781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11801.695358] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11807.232946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11808.682799] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [11816.883501] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11823.778814] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11824.072421] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11933.163130] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.190713] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11933.195302] Lustre: Skipped 1 previous similar message [11936.436863] Lustre: server umount lustre-OST0001 complete [11938.277678] LustreError: 334610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11938.301588] LustreError: 334610:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11943.402980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11943.412480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11943.414086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11943.414091] Lustre: Skipped 1 previous similar message [11948.421942] Lustre: server umount lustre-OST0000 complete [11953.633800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11953.635273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11953.644911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11953.644919] Lustre: Skipped 1 previous similar message [11953.665854] Lustre: Skipped 2 previous similar messages [11958.354849] Lustre: server umount lustre-MDT0000 complete [11958.753582] LustreError: 331642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11958.771706] LustreError: 331642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11961.963205] LustreError: 331624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722500 with bad export cookie 16004400495647858383 [11961.966670] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11961.973482] LustreError: 331624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11962.289954] Lustre: server umount lustre-MDT0001 complete [11972.595584] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [11975.090639] Key type lgssc unregistered [11975.467429] LNet: 336583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11975.478112] LNetError: 336583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11975.511609] LNet: Removed LNI 192.168.203.122@tcp [11976.504063] Key type .llcrypt unregistered [11976.508668] Key type ._llcrypt unregistered [12020.826933] Key type ._llcrypt registered [12020.832598] Key type .llcrypt registered [12020.970205] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12022.355041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12022.385214] alg: No test for adler32 (adler32-zlib) [12023.455127] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [12023.743065] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [12025.408205] Key type lgssc registered [12026.583539] Lustre: Echo OBD driver; http://www.lustre.org/ [12035.840222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12035.872561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12037.541467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12042.490759] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12047.843692] LustreError: 338891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12047.859829] LustreError: 338891:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12051.132853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12051.547219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12057.142802] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12060.188746] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12067.494730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12067.945054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12074.405687] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12076.150892] LustreError: 340383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12081.651053] LustreError: 340384:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.692540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12083.793320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12084.170260] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12089.339798] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12091.173270] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12100.635614] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12108.140689] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12120.039048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12120.044510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12120.067522] Lustre: Skipped 2 previous similar messages [12120.072262] Lustre: Skipped 1 previous similar message [12125.163814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12125.169537] Lustre: Skipped 5 previous similar messages [12125.720556] Lustre: server umount lustre-MDT0000 complete [12129.370266] LustreError: 340029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722667 with bad export cookie 8829950876954164610 [12129.373371] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12129.383249] LustreError: 340029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12129.654834] Lustre: server umount lustre-MDT0001 complete [12143.134938] Lustre: server umount lustre-OST0000 complete [12146.656185] Lustre: 338143:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790722668/real 1790722668] req@ffff9a9c0f9f7480 x1877708700187520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790722684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12146.686152] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12146.695182] Lustre: Skipped 1 previous similar message [12147.423171] Lustre: server umount lustre-OST0001 complete [12164.222109] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12174.732754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12175.151233] LustreError: 344153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12175.173773] LustreError: 344153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12175.234875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12179.859656] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12180.451207] LustreError: 344154:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12188.620064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12189.071813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12194.014200] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12196.910596] Lustre: 345293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12203.600862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12210.541916] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12214.250799] LustreError: 345645:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12214.271789] LustreError: 345645:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12220.168694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12220.402396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12220.505619] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12220.511248] Lustre: Skipped 1 previous similar message [12225.546357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12227.414738] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12236.071880] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12239.857404] Lustre: 347163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12245.486035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12245.494455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.527345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12250.689589] Lustre: server umount lustre-MDT0000 complete [12251.116778] LustreError: 345668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12251.138521] LustreError: 345668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12254.739742] LustreError: 344135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790722792 with bad export cookie 8829950876954166311 [12254.768085] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12255.212833] Lustre: server umount lustre-MDT0001 complete [12269.650733] Lustre: server umount lustre-OST0000 complete [12272.618644] Lustre: 338145:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790722794/real 1790722794] req@ffff9a9d1024c700 x1877708700235904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790722810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12272.641613] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12272.657159] Lustre: Skipped 3 previous similar messages [12273.609442] Lustre: server umount lustre-OST0001 complete [12277.084913] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12278.626431] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12280.391563] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 19:00:17 (1790722817) [12303.342146] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12313.114136] loop0: detected capacity change from 0 to 1099511627776 [12313.735222] SGI XFS with ACLs, security attributes, quota, no debug enabled [12313.744978] *** VALIDATE xfs *** [12313.774120] XFS (loop0): Mounting V5 Filesystem [12313.793536] XFS (loop0): Ending clean mount [12315.014639] loop1: detected capacity change from 0 to 18691697672192 [12413.501441] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [12425.391460] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12462.412636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12462.691640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12462.723111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12462.813692] Lustre: lustre-MDT0000: new disk, initializing [12462.869984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12462.884978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12467.264127] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12477.670808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12477.817826] Lustre: 352306:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12477.872320] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12477.874934] Lustre: Skipped 1 previous similar message [12477.936374] Lustre: lustre-MDT0001: new disk, initializing [12478.033204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12478.051502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12482.094948] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12491.288311] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12496.485245] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12503.811233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12504.070389] Lustre: lustre-OST0000: new disk, initializing [12504.081979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12504.091515] Lustre: 354275:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12505.412471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12505.423463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12505.616091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12511.400172] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12521.208116] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12525.183198] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12525.355674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12530.125707] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12530.358927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12536.293584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12536.294524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12536.295232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12536.295237] Lustre: Skipped 3 previous similar messages [12536.313144] Lustre: Skipped 1 previous similar message [12538.755939] Lustre: server umount lustre-OST0000 complete [12544.484940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12544.501541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12544.530295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12544.539588] Lustre: Skipped 1 previous similar message [12556.769785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12556.780240] Lustre: Skipped 5 previous similar messages [12557.281508] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12557.645312] Lustre: server umount lustre-MDT0000 complete [12561.892840] LustreError: 352313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12561.938642] LustreError: 352313:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12562.869987] LustreError: 352299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790723100 with bad export cookie 8829950876954167865 [12562.880954] LustreError: 352299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12562.884348] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12563.267287] Lustre: server umount lustre-MDT0001 complete [12573.862499] XFS (loop0): Unmounting Filesystem [12579.166146] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 19:05:16 (1790723116) [12588.051204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12588.790557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12588.803693] Lustre: Skipped 2 previous similar messages [12593.525510] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12601.405375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12606.168056] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12614.224992] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12620.484360] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12627.248032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12634.421218] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12641.634501] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12654.566722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12654.579906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12654.592141] Lustre: Skipped 1 previous similar message [12654.600882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12658.752203] Lustre: server umount lustre-OST0000 complete [12663.265827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12663.274624] LustreError: Skipped 1 previous similar message [12668.803993] Lustre: server umount lustre-MDT0000 complete [12671.972169] LustreError: 356834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12671.980189] LustreError: 356834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12672.708295] LustreError: 358681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790723210 with bad export cookie 8829950876954168747 [12672.715884] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12672.721184] LustreError: 358681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12673.100742] Lustre: server umount lustre-MDT0001 complete [12685.220321] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [12688.788398] Key type lgssc unregistered [12689.131898] LNet: 360468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12689.142355] LNetError: 360468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12689.166883] LNet: Removed LNI 192.168.203.122@tcp [12690.320635] Key type .llcrypt unregistered [12690.326553] Key type ._llcrypt unregistered [12717.504886] Key type ._llcrypt registered [12717.507463] Key type .llcrypt registered [12717.606148] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [12718.463369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12718.596869] alg: No test for adler32 (adler32-zlib) [12719.602707] Lustre: Lustre: Build Version: 2.17.59_2_g598166e [12719.857200] LNet: Added LNI 192.168.203.122@tcp [8/256/0/180] [12721.529283] Key type lgssc registered [12722.585679] Lustre: Echo OBD driver; http://www.lustre.org/ [12730.492995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [12730.516943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12732.222844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12737.952913] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12742.636259] LustreError: 362054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12742.665484] LustreError: 362054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12747.746563] LustreError: 362053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12749.480297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12750.014253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12754.584713] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12762.838352] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12769.056310] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12776.730276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12777.174330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12783.656817] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12791.337080] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12804.569932] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.22@tcp (stopping) [12805.093510] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12805.108411] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12805.114913] Lustre: Skipped 1 previous similar message [12808.512936] Lustre: server umount lustre-MDT0001 complete [12809.658278] LustreError: 362050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12814.777803] LustreError: 363893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12814.807728] LustreError: 363893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12818.791972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12818.880302] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12818.884400] Lustre: Found index 1 for lustre-MDT0001, updating log [12818.912286] Lustre: 362041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12819.248527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12822.992406] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [12824.558735] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12826.617146] LustreError: 365654:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12826.623774] LustreError: 365654:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12826.634721] LustreError: 362045:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12826.744714] LustreError: 365657:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.122@tcp: cfg command failed: rc = -17 [12826.770677] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12826.770677] [13131.747663] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13131.749702] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13131.758426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13131.775413] Lustre: Skipped 2 previous similar messages [13134.679286] Lustre: server umount lustre-MDT0001 complete [13136.869638] LustreError: 362053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13136.879986] LustreError: 362053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13146.203943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13146.305338] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13146.316445] Lustre: Found index 1 for lustre-MDT0001, updating log [13146.345307] Lustre: 362042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13146.770145] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13151.325869] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13152.240599] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13154.299301] LustreError: 374068:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13154.329758] LustreError: 374068:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13154.348316] LustreError: 362045:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13154.363144] LustreError: 362045:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13154.420095] LustreError: 374084:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13154.431295] LustreError: 374084:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13154.439590] LustreError: 374084:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.122@tcp: cfg command failed: rc = -17 [13154.449270] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.122@tcp [13154.449270] [13459.425974] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13459.437550] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13459.445527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13459.448572] Lustre: Skipped 1 previous similar message [13459.487532] Lustre: Skipped 3 previous similar messages [13463.534207] Lustre: server umount lustre-MDT0001 complete [13464.553816] LustreError: 363893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13464.592368] LustreError: 363893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13474.993385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13475.171157] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13475.175655] Lustre: Found index 1 for lustre-MDT0001, updating log [13475.214310] Lustre: 362042:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13475.637225] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13480.814963] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13480.957957] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13483.046261] LustreError: 382560:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13483.055401] LustreError: 382560:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13483.074436] LustreError: 362045:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13483.086924] LustreError: 362045:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13483.221115] LustreError: 382568:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13483.232463] LustreError: 382568:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.122@tcp: cfg command failed: rc = -114 [13483.238801] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13483.238801] [13808.612021] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13808.615339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13808.641560] Lustre: Skipped 2 previous similar messages [13808.657900] Lustre: Skipped 3 previous similar messages [13810.165527] Lustre: server umount lustre-MDT0000 complete [13813.731606] LustreError: 365770:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13813.745790] LustreError: 365770:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [13814.394668] LustreError: 365238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790724352 with bad export cookie 13897745760803145073 [13814.415728] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13818.854168] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13818.892458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13820.794090] Lustre: server umount lustre-MDT0001 complete [13825.276546] Lustre: server umount lustre-OST0000 complete [13835.877313] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [13846.909589] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [13887.496861] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 19:27:04 (1790724424) [13896.040234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13896.533172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13896.576128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13896.708176] Lustre: lustre-MDT0000: new disk, initializing [13896.836272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13896.892256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13902.802684] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13915.564748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13915.682289] Lustre: 393085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13915.736367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13915.740993] Lustre: Skipped 1 previous similar message [13915.798594] Lustre: lustre-MDT0001: new disk, initializing [13915.908830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13915.967842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13915.984303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13921.861911] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13931.180405] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13937.724717] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13944.861205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13945.265201] Lustre: lustre-OST0000: new disk, initializing [13945.268429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13945.279878] Lustre: 395057:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13945.381807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13952.143337] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_default_debug -1 all [13952.577270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13952.599251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13952.751584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13962.231726] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13970.932315] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13970.947081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13970.961654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13970.964345] Lustre: Skipped 1 previous similar message [13972.482267] Lustre: server umount lustre-MDT0000 complete [13973.473868] LustreError: 393098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13973.494073] LustreError: 393098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13976.994950] LustreError: 393079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790724515 with bad export cookie 13897745760803147033 [13977.007159] LustreError: MGC192.168.203.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13977.017160] LustreError: 393079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13977.018349] LustreError: 393893:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13977.432830] Lustre: server umount lustre-MDT0001 complete [13991.052141] Lustre: server umount lustre-OST0000 complete [14022.989047] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing set_hostid [14035.114888] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing load_modules_local [14069.429996] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13759 sec ========= 19:30:06 (1790724606) [14071.369671] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:30:08 (1790724608) === [14075.591545] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:30:12 (1790724612) === [14119.771156] Lustre: DEBUG MARKER: oleg322-server.virtnet: executing unload_modules_local [14122.652870] Key type lgssc unregistered [14123.027381] LNet: 401424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14123.036945] LNetError: 401424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14123.075092] LNet: Removed LNI 192.168.203.122@tcp [14124.129421] Key type .llcrypt unregistered [14124.135720] Key type ._llcrypt unregistered