[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 516327903 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001035] APIC: Switch to symmetric I/O mode setup [ 0.002392] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007026] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009130] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.011050] SELinux: Initializing. [ 0.012087] *** VALIDATE selinux *** [ 0.020626] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025699] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026170] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027188] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028181] *** VALIDATE tmpfs *** [ 0.029490] *** VALIDATE proc *** [ 0.031413] *** VALIDATE cgroup *** [ 0.032028] *** VALIDATE cgroup2 *** [ 0.033359] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037035] Spectre V2 : User space: Vulnerable [ 0.038018] Speculative Store Bypass: Vulnerable [ 0.041727] debug: unmapping init [mem 0xffffffffa6859000-0xffffffffa6860fff] [ 0.043662] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045453] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046067] ... version: 2 [ 0.047028] ... bit width: 48 [ 0.048018] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051023] ... fixed-purpose events: 3 [ 0.052016] ... event mask: 000000070000000f [ 0.053384] rcu: Hierarchical SRCU implementation. [ 0.056896] smp: Bringing up secondary CPUs ... [ 0.057695] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.069170] smp: Brought up 1 node, 4 CPUs [ 0.071014] smpboot: Max logical packages: 1 [ 0.072019] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.113053] node 0 deferred pages initialised in 39ms [ 0.118178] devtmpfs: initialized [ 0.119278] x86/mm: Memory block size: 128MB [ 0.121265] gcov: version magic: 0x41383552 [ 0.123309] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.124128] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.125218] pinctrl core: initialized pinctrl subsystem [ 0.126163] [ 0.126686] ************************************************************* [ 0.127010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.128013] ** ** [ 0.129014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.130011] ** ** [ 0.131017] ** This means that this kernel is built to expose internal ** [ 0.132017] ** IOMMU data structures, which may compromise security on ** [ 0.133013] ** your system. ** [ 0.134029] ** ** [ 0.135014] ** If you see this message and you are not debugging the ** [ 0.136016] ** kernel, report this immediately to your vendor! ** [ 0.137015] ** ** [ 0.138014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139014] ************************************************************* [ 0.140778] NET: Registered protocol family 16 [ 0.141415] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.142059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.143053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.144412] cpuidle: using governor menu [ 0.146372] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.148824] PCI: Using configuration type 1 for base access [ 0.152186] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.162063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.164121] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.168075] cryptd: max_cpu_qlen set to 1000 [ 0.170187] ACPI: Added _OSI(Module Device) [ 0.171012] ACPI: Added _OSI(Processor Device) [ 0.172012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.173012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.177598] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.181303] ACPI: Interpreter enabled [ 0.182080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.183017] ACPI: Using IOAPIC for interrupt routing [ 0.184166] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.188490] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.202484] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.205151] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.209029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.213158] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.220263] acpiphp: Slot [2] registered [ 0.222253] acpiphp: Slot [5] registered [ 0.224193] acpiphp: Slot [6] registered [ 0.225176] acpiphp: Slot [7] registered [ 0.227098] acpiphp: Slot [8] registered [ 0.228111] acpiphp: Slot [9] registered [ 0.230087] acpiphp: Slot [10] registered [ 0.232107] acpiphp: Slot [3] registered [ 0.233106] acpiphp: Slot [4] registered [ 0.235160] acpiphp: Slot [11] registered [ 0.237102] acpiphp: Slot [12] registered [ 0.239099] acpiphp: Slot [13] registered [ 0.241184] acpiphp: Slot [14] registered [ 0.243107] acpiphp: Slot [15] registered [ 0.244106] acpiphp: Slot [16] registered [ 0.246207] acpiphp: Slot [17] registered [ 0.248180] acpiphp: Slot [18] registered [ 0.250078] acpiphp: Slot [19] registered [ 0.251086] acpiphp: Slot [20] registered [ 0.253105] acpiphp: Slot [21] registered [ 0.255192] acpiphp: Slot [22] registered [ 0.256091] acpiphp: Slot [23] registered [ 0.258213] acpiphp: Slot [24] registered [ 0.259116] acpiphp: Slot [25] registered [ 0.261102] acpiphp: Slot [26] registered [ 0.262256] acpiphp: Slot [27] registered [ 0.264104] acpiphp: Slot [28] registered [ 0.266153] acpiphp: Slot [29] registered [ 0.267186] acpiphp: Slot [30] registered [ 0.269128] acpiphp: Slot [31] registered [ 0.271085] PCI host bridge to bus 0000:00 [ 0.273051] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.276029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.284021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.289213] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293641] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.297487] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.308776] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.314059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.317018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.320017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327449] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330978] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.337963] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.357017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.363109] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.370606] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.417015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.436000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.476023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.515816] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.527018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.563041] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.587012] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.595015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.603000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.629021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.643593] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.652022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.660018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.679029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.690828] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.699034] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.706233] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.752021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.783206] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.786893] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789973] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.792517] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.795562] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.801159] iommu: Default domain type: Passthrough [ 0.806369] SCSI subsystem initialized [ 0.808245] ACPI: bus type USB registered [ 0.810106] usbcore: registered new interface driver usbfs [ 0.813106] usbcore: registered new interface driver hub [ 0.815128] usbcore: registered new device driver usb [ 0.818142] pps_core: LinuxPPS API ver. 1 registered [ 0.820011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.823101] PTP clock support registered [ 0.826281] EDAC MC: Ver: 3.0.0 [ 0.828304] PCI: Using ACPI for IRQ routing [ 0.831180] NetLabel: Initializing [ 0.833012] NetLabel: domain hash size = 128 [ 0.835013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.837075] NetLabel: unlabeled traffic allowed by default [ 0.839328] vgaarb: loaded [ 0.841320] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.843012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.851895] clocksource: Switched to clocksource kvm-clock [ 0.977692] VFS: Disk quotas dquot_6.6.0 [ 0.979520] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.982449] *** VALIDATE ramfs *** [ 0.983838] *** VALIDATE hugetlbfs *** [ 0.986659] pnp: PnP ACPI init [ 0.989486] pnp: PnP ACPI: found 6 devices [ 1.006695] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.010648] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.013458] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.015863] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.018858] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.021825] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.025395] NET: Registered protocol family 2 [ 1.028342] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.033794] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.038213] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.044067] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.047808] TCP: Hash tables configured (established 65536 bind 65536) [ 1.050997] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.054764] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.058479] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.061808] NET: Registered protocol family 1 [ 1.064768] RPC: Registered named UNIX socket transport module. [ 1.067349] RPC: Registered udp transport module. [ 1.069080] RPC: Registered tcp transport module. [ 1.070818] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.073325] NET: Registered protocol family 44 [ 1.074765] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.077254] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.079663] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.082369] PCI: CLS 0 bytes, default 64 [ 1.083993] Unpacking initramfs... [ 2.921030] debug: unmapping init [mem 0xffff98dcbcc54000-0xffff98dcbffbffff] [ 2.924240] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.926524] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.929139] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.927329] Initialise system trusted keyrings [ 3.928500] Key type blacklist registered [ 3.930148] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.939485] zbud: loaded [ 3.944472] *** VALIDATE nfs *** [ 3.945865] *** VALIDATE nfs4 *** [ 3.947432] pstore: using deflate compression [ 3.951613] Platform Keyring initialized [ 4.101518] NET: Registered protocol family 38 [ 4.104156] Key type asymmetric registered [ 4.105218] Asymmetric key parser 'x509' registered [ 4.106789] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.108869] io scheduler mq-deadline registered [ 4.110475] io scheduler kyber registered [ 4.111593] io scheduler bfq registered [ 4.113270] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.116601] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.129394] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.151669] ACPI: Power Button [PWRF] [ 4.171376] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.186617] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.210712] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.220258] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.256173] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.288045] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.318307] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.323655] Non-volatile memory driver v1.3 [ 4.325472] Linux agpgart interface v0.103 [ 4.359952] virtio_blk virtio1: [vda] 139504 512-byte logical blocks (71.4 MB/68.1 MiB) [ 4.363186] vda: detected capacity change from 0 to 71426048 [ 4.387356] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.389800] vdb: detected capacity change from 0 to 1073741824 [ 4.462517] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.467873] vdc: detected capacity change from 0 to 2621440000 [ 4.495373] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.500933] vdd: detected capacity change from 0 to 2621440000 [ 4.525362] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.527862] vde: detected capacity change from 0 to 4294967296 [ 4.555288] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.559350] vdf: detected capacity change from 0 to 4294967296 [ 4.576345] libphy: Fixed MDIO Bus: probed [ 4.582241] usbcore: registered new interface driver usbserial_generic [ 4.589151] usbserial: USB Serial support registered for generic [ 4.593709] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.601288] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.604508] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.608858] mousedev: PS/2 mouse device common for all mice [ 4.611661] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.613973] rtc_cmos 00:05: RTC can wake from S4 [ 4.618894] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.624432] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.629680] rtc_cmos 00:05: registered as rtc0 [ 4.631639] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.634536] intel_pstate: CPU model not supported [ 4.638466] hid: raw HID events driver (C) Jiri Kosina [ 4.640915] usbcore: registered new interface driver usbhid [ 4.643362] usbhid: USB HID core driver [ 4.644990] drop_monitor: Initializing network drop monitor service [ 4.647701] Initializing XFRM netlink socket [ 4.649855] NET: Registered protocol family 10 [ 4.653315] Segment Routing with IPv6 [ 4.654805] NET: Registered protocol family 17 [ 4.656760] mpls_gso: MPLS GSO support [ 4.663038] RAS: Correctable Errors collector initialized. [ 4.665194] AVX version of gcm_enc/dec engaged. [ 4.666487] AES CTR mode by8 optimization enabled [ 4.767217] sched_clock: Marking stable (4767050802, 0)->(5979253206, -1212202404) [ 4.771925] registered taskstats version 1 [ 4.774777] Loading compiled-in X.509 certificates [ 4.779992] zswap: loaded using pool lzo/zbud [ 4.818763] Key type big_key registered [ 4.872298] Key type encrypted registered [ 4.875376] ima: No TPM chip found, activating TPM-bypass! [ 4.878387] ima: Allocated hash algorithm: sha1 [ 4.880452] ima: No architecture policies found [ 4.883304] evm: Initialising EVM extended attributes: [ 4.886024] evm: security.selinux [ 4.888437] evm: security.ima [ 4.889668] evm: security.capability [ 4.893240] evm: HMAC attrs: 0x1 [ 4.895599] rtc_cmos 00:05: setting system clock to 2026-06-26 14:41:39 UTC (1782484899) [ 4.902653] debug: unmapping init [mem 0xffffffffa7803000-0xffffffffa79fffff] [ 4.909639] debug: unmapping init [mem 0xffffffffa6582000-0xffffffffa6858fff] [ 4.919162] Write protecting the kernel read-only data: 28672k [ 4.922815] debug: unmapping init [mem 0xffffffffa4c03000-0xffffffffa4dfffff] [ 4.925769] debug: unmapping init [mem 0xffffffffa5514000-0xffffffffa55fffff] [ 4.975715] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.985332] systemd[1]: Detected virtualization kvm. [ 4.987762] systemd[1]: Detected architecture x86-64. [ 4.989828] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.032368] systemd[1]: No hostname configured. [ 5.035286] systemd[1]: Set hostname to . [ 5.037942] random: systemd: uninitialized urandom read (16 bytes read) [ 5.040668] systemd[1]: Initializing machine ID from random generator. [ 5.091580] random: ln: uninitialized urandom read (6 bytes read) [ 5.502788] random: systemd: uninitialized urandom read (16 bytes read) [ 5.510973] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.651532] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.658289] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.175428] device-mapper: uevent: version 1.0.3 [ 7.179450] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.892146] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 9.122779] random: fast init done [ 9.545389] scsi host0: ata_piix [ 9.581216] scsi host1: ata_piix [ 9.582760] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.584844] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.818692] random: crng init done [ 14.854196] random: 7 urandom warning(s) missed due to ratelimiting [ 16.706514] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.116580] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.615576] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.553793] SELinux: Disabled at runtime. [ 24.010407] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.021404] systemd[1]: Detected virtualization kvm. [ 24.023469] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.598893] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.603405] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.616130] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.619836] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.623183] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.632201] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.643163] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 26.350948] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 28.533519] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.863380] hrtimer: interrupt took 20389029 ns [ 33.026123] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.073082] input: PC Speaker as /devices/platform/pcspkr/input/input5 [* ] A start job is running for Configur…only root support (10s / no limit)[ 36.162904] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [** ] A start job is running for Configur…only root support (11s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit)[ 39.071705] EDAC sbridge: Ver: 1.1.2 [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ *] A start job is running for Configur…only root support (15s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root support (18s / no limit)[ 43.706666] Key type dns_resolver registered [* ] A start job is running for Configur…only root support (18s / no limit) [** ] A start job is running for Configur…only root support (19s / no limit) [*** ] A start job is running for Configur…only root support (19s / no limit)[ 45.102402] NFS: Registering the id_resolver key type [ 45.109735] Key type id_resolver registered [ 45.114688] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (20s / no limit) [ *** ] A start job is running for Configur…only root support (20s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg454-server login: [ 105.851028] libcfs: loading out-of-tree module taints kernel. [ 105.893273] Key type ._llcrypt registered [ 105.895376] Key type .llcrypt registered [ 105.983030] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 118.893952] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 119.905346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 119.913828] alg: No test for adler32 (adler32-zlib) [ 121.195854] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 121.715605] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 123.399169] Key type lgssc registered [ 124.453821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.134389] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.731656] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 175.158107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 175.182305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.385411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.416112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.480669] Lustre: lustre-MDT0000: new disk, initializing [ 176.539826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.553228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 179.566256] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.867789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.948406] Lustre: 6515:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.971444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.976139] Lustre: Skipped 1 previous similar message [ 190.033463] Lustre: lustre-MDT0001: new disk, initializing [ 190.081721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.115526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.125423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 193.224245] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.902574] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 203.509625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.685707] Lustre: lustre-OST0000: new disk, initializing [ 203.690796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.698876] Lustre: 8453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.767873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.146453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.153821] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.236997] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 208.139665] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.742708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.836741] Lustre: lustre-OST0001: new disk, initializing [ 218.840715] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 218.845919] Lustre: 9525:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.889763] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.320558] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.312684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 226.320053] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 226.374876] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.954882] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.810776] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 239.248387] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing check_logdir /tmp/testlogs/ [ 242.301949] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing yml_node [ 245.398980] Lustre: DEBUG MARKER: Client: 2.17.54.4 [ 247.160536] Lustre: DEBUG MARKER: MDS: 2.17.54.4 [ 248.918026] Lustre: DEBUG MARKER: OSS: 2.17.54.4 [ 250.009350] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 26 10:45:43 EDT 2026 [ 260.907782] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 261.934547] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 277.477113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 277.484940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.497139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.500145] Lustre: Skipped 3 previous similar messages [ 281.290745] Lustre: server umount lustre-MDT0000 complete [ 283.479399] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485178 with bad export cookie 18285806774282328811 [ 283.480283] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.489108] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 287.711716] LustreError: 8491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.714935] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.715974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 287.725423] LustreError: 8491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 287.733915] Lustre: Skipped 4 previous similar messages [ 288.737519] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 288.737536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 288.762802] Lustre: Skipped 2 previous similar messages [ 289.763455] Lustre: server umount lustre-MDT0001 complete [ 292.391878] Lustre: server umount lustre-OST0000 complete [ 294.872492] Lustre: server umount lustre-OST0001 complete [ 299.038602] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 304.395360] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 328.023767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 328.234941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 328.259684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 328.314649] Lustre: lustre-MDT0000: new disk, initializing [ 328.393872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 328.407250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 331.688095] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 339.973943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.088603] Lustre: 15070:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 340.132248] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 340.137746] Lustre: Skipped 1 previous similar message [ 340.221564] Lustre: lustre-MDT0001: new disk, initializing [ 340.314293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 340.337526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 340.347137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 343.380475] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 350.118363] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 354.293158] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 359.066780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 359.294224] Lustre: lustre-OST0000: new disk, initializing [ 359.304362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 359.310519] Lustre: 17040:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 359.391723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.382873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 360.398874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 360.483066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 363.918795] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 369.885295] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 372.520758] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 372.675651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 375.231043] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 375.349803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 380.898539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 380.898791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 380.905534] LustreError: Skipped 1 previous similar message [ 380.906422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 380.914133] Lustre: Skipped 1 previous similar message [ 380.922940] Lustre: Skipped 1 previous similar message [ 382.677400] Lustre: server umount lustre-OST0000 complete [ 386.019630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.022792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 386.028638] Lustre: Skipped 1 previous similar message [ 386.038097] Lustre: Skipped 1 previous similar message [ 396.257146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 396.262364] Lustre: Skipped 3 previous similar messages [ 399.327184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 399.430966] Lustre: server umount lustre-MDT0000 complete [ 401.387656] LustreError: 15076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.411375] LustreError: 15076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 402.057797] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485296 with bad export cookie 18285806774282330351 [ 402.063237] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 402.071638] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 402.408241] Lustre: server umount lustre-MDT0001 complete [ 406.083058] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 10:48:19 (1782485299) [ 410.637566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.905194] LustreError: 19303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.960984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.125477] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 416.224620] LustreError: 19304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 419.591186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.510508] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 427.457866] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 430.824402] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 435.083702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 438.931222] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 443.350324] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 449.885654] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 455.138441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.142260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 455.146734] Lustre: Skipped 2 previous similar messages [ 455.151951] Lustre: Skipped 2 previous similar messages [ 460.481143] Lustre: server umount lustre-MDT0000 complete [ 462.405267] LustreError: 21830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485357 with bad export cookie 18285806774282331233 [ 462.406151] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.411038] LustreError: 21830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 462.563175] Lustre: server umount lustre-MDT0001 complete [ 474.685902] Lustre: server umount lustre-OST0000 complete [ 487.301601] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 493.191424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.423452] LustreError: 23679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.461397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 493.466433] Lustre: Skipped 2 previous similar messages [ 495.686924] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 500.504933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.051290] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 504.721611] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 508.380408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 512.266753] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 517.439585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 517.571696] Lustre: lustre-OST0001: new disk, initializing [ 517.574703] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 517.579475] Lustre: 25992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 518.710095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 518.718586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 518.806151] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 521.914808] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 528.466728] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 530.562698] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 539.623791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.626983] LustreError: 23675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.629434] Lustre: Skipped 2 previous similar messages [ 539.641944] LustreError: 23675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 539.693646] Lustre: server umount lustre-MDT0000 complete [ 541.636188] LustreError: 25991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485436 with bad export cookie 18285806774282332080 [ 541.642114] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 541.644571] LustreError: 25991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 543.890795] Lustre: server umount lustre-MDT0001 complete [ 555.600839] Lustre: server umount lustre-OST0000 complete [ 568.072931] Lustre: server umount lustre-OST0001 complete [ 576.874306] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 582.850770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.176597] LustreError: 29010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.192833] LustreError: 29010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 585.696124] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 590.582246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.314078] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 595.052910] Lustre: 30157:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 599.111419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 602.797618] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 607.522756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 610.785242] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 615.138454] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 617.302623] Lustre: 32035:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 621.537022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 621.541821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.550442] Lustre: Skipped 1 previous similar message [ 621.554183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 621.558103] Lustre: Skipped 3 previous similar messages [ 626.529538] LustreError: 32236:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (ffff98dd3ee25600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 626.568964] Lustre: server umount lustre-MDT0000 complete [ 628.612270] LustreError: 28992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485523 with bad export cookie 18285806774282333375 [ 628.620082] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 628.625642] LustreError: 28992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 628.807357] Lustre: server umount lustre-MDT0001 complete [ 641.175034] Lustre: server umount lustre-OST0000 complete [ 653.322912] Lustre: server umount lustre-OST0001 complete [ 658.281943] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 10:52:31 (1782485551) [ 673.971589] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 679.161315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.389810] LustreError: 34834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.401199] LustreError: 34834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 679.443642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 679.447381] Lustre: Skipped 7 previous similar messages [ 681.867679] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 687.460103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.586448] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 692.557296] Lustre: 35981:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 697.046230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 700.562829] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 705.574909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 709.079533] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 713.727559] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 715.575181] Lustre: 37857:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 718.304588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 718.311699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 718.320692] Lustre: Skipped 3 previous similar messages [ 718.327033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 718.335165] Lustre: Skipped 4 previous similar messages [ 723.697371] Lustre: server umount lustre-MDT0000 complete [ 725.325444] LustreError: 35583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485619 with bad export cookie 18285806774282334481 [ 725.329056] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.332844] LustreError: 35583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 725.479887] Lustre: server umount lustre-MDT0001 complete [ 736.516196] Lustre: server umount lustre-OST0000 complete [ 748.907351] Lustre: server umount lustre-OST0001 complete [ 753.979303] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 10:54:07 (1782485647) [ 762.287298] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 768.046638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.729391] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 775.464557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.098702] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 779.913515] Lustre: 41147:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 784.019692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 788.018115] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 797.273725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 806.768643] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 815.733665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 820.612202] Lustre: 43025:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 821.866221] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 821.871513] Lustre: Skipped 1 previous similar message [ 826.854499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 826.863194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.871225] Lustre: Skipped 3 previous similar messages [ 832.025571] Lustre: server umount lustre-MDT0000 complete [ 833.510064] LustreError: 39998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.542423] LustreError: 39998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 837.459762] LustreError: 39980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485732 with bad export cookie 18285806774282335580 [ 837.463418] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.482603] LustreError: 39980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 844.108904] Lustre: server umount lustre-MDT0001 complete [ 855.519257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.207887] LustreError: 3653:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98dd0735c700 x1869071008478720/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 869.322778] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 873.966280] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 880.462369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.120186] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 886.267063] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 886.282425] Lustre: Skipped 1 previous similar message [ 892.737291] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 894.081861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 898.900985] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 900.464120] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 906.725071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 906.728686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 906.736334] Lustre: Skipped 18 previous similar messages [ 911.196038] Lustre: server umount lustre-MDT0000 complete [ 914.286115] LustreError: 40748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485808 with bad export cookie 18285806774282336707 [ 914.291404] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 914.295625] LustreError: 40748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 914.576804] Lustre: server umount lustre-MDT0001 complete [ 928.453199] Lustre: server umount lustre-OST0000 complete [ 942.327853] Lustre: server umount lustre-OST0001 complete [ 949.337778] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 10:57:22 (1782485842) [ 963.524553] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 971.398854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.884935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 971.892759] Lustre: Skipped 9 previous similar messages [ 975.049541] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 982.480346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.173852] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 989.667238] Lustre: 48788:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 995.632984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1001.592087] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1009.685307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1016.794571] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1024.566484] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1029.218803] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1040.870914] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.884172] Lustre: Skipped 11 previous similar messages [ 1042.835888] Lustre: server umount lustre-MDT0000 complete [ 1046.347357] LustreError: 47617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485940 with bad export cookie 18285806774282337491 [ 1046.353489] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1046.357379] LustreError: 47617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1046.696124] Lustre: server umount lustre-MDT0001 complete [ 1061.445214] Lustre: server umount lustre-OST0000 complete [ 1076.303517] Lustre: server umount lustre-OST0001 complete [ 1087.175359] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 10:59:39 (1782485979) [ 1089.674622] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1091.517478] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 10:59:43 (1782485983) [ 1104.169710] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1112.473317] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1136.010151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1136.305953] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1136.368444] Lustre: lustre-MDT0000: new disk, initializing [ 1136.461117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1141.165867] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1152.611396] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1161.137072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1161.573593] Lustre: 54859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1162.862102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1162.908223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1168.344823] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1177.054921] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1179.169611] Lustre: DEBUG MARKER: create test files [ 1203.683991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1203.697946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1203.705028] Lustre: Skipped 4 previous similar messages [ 1206.339773] Lustre: server umount lustre-OST0000 complete [ 1209.820837] Lustre: server umount lustre-MDT0000 complete [ 1218.083462] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1220.249231] Key type lgssc unregistered [ 1220.484059] LNet: 56502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.490642] LNetError: 56502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.507783] LNet: Removed LNI 192.168.204.154@tcp [ 1221.256162] Key type .llcrypt unregistered [ 1221.257966] Key type ._llcrypt unregistered [ 1240.722420] Key type ._llcrypt registered [ 1240.725184] Key type .llcrypt registered [ 1240.813642] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1241.331141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1241.347238] alg: No test for adler32 (adler32-zlib) [ 1242.302726] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 1242.471091] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1244.111119] Key type lgssc registered [ 1244.864392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1250.288479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1250.303845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.589156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1254.621418] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1259.141418] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1263.505664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1263.772664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.534908] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1272.494889] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1277.039394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:100 to 0x240000400:129) [ 1277.118717] Lustre: 59880:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1277.124490] Lustre: 59880:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1277.128929] Lustre: 59880:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1277.133222] Lustre: 59880:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1277.137444] Lustre: 59880:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1277.141235] Lustre: 59880:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1281.671522] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1285.164062] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1292.768494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1292.772496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.782250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1294.924810] Lustre: server umount lustre-OST0000 complete [ 1297.674607] Lustre: server umount lustre-MDT0000 complete [ 1303.921719] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1305.845861] Key type lgssc unregistered [ 1306.088806] LNet: 60823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.094750] LNetError: 60823:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.109346] LNet: Removed LNI 192.168.204.154@tcp [ 1306.680163] Key type .llcrypt unregistered [ 1306.682099] Key type ._llcrypt unregistered [ 1323.805473] Key type ._llcrypt registered [ 1323.807458] Key type .llcrypt registered [ 1323.880789] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1324.434503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.444938] alg: No test for adler32 (adler32-zlib) [ 1325.358941] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 1325.494277] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1327.119173] Key type lgssc registered [ 1327.833283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.175300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.194390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.525039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.159401] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1340.660365] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1344.696086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1344.888296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.408569] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1353.209836] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1359.148821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1359.195660] Lustre: 63637:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1359.203221] Lustre: 63637:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1359.207343] Lustre: 63637:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1359.212376] Lustre: 63637:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1359.216810] Lustre: 63637:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1359.221138] Lustre: 63637:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1363.155898] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1369.572323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1369.573345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.586411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1374.691845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1375.079638] Lustre: server umount lustre-OST0000 complete [ 1377.825508] Lustre: server umount lustre-MDT0000 complete [ 1383.690516] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 1385.379322] Key type lgssc unregistered [ 1385.595779] LNet: 65194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.600881] LNetError: 65194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.610805] LNet: Removed LNI 192.168.204.154@tcp [ 1386.114416] Key type .llcrypt unregistered [ 1386.116462] Key type ._llcrypt unregistered [ 1398.017901] Key type ._llcrypt registered [ 1398.021399] Key type .llcrypt registered [ 1398.095108] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1409.220490] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1409.960191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.045285] alg: No test for adler32 (adler32-zlib) [ 1411.094428] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 1411.272361] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 1412.935420] Key type lgssc registered [ 1413.856099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1437.012117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1437.036457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.261992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1438.289542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1438.371349] Lustre: lustre-MDT0000: new disk, initializing [ 1438.435251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.454806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1441.504110] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1448.890635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.990561] Lustre: 68322:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1449.031639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1449.034906] Lustre: Skipped 1 previous similar message [ 1449.084045] Lustre: lustre-MDT0001: new disk, initializing [ 1449.156435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1449.189730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1449.203241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1452.019294] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1457.795533] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1460.939851] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1465.008295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1465.170763] Lustre: lustre-OST0000: new disk, initializing [ 1465.174346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1465.179303] Lustre: 70289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1465.237985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.690758] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1471.009219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1471.016603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1471.124946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1475.281527] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1478.349299] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1478.496862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1481.718551] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1481.887670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1486.308947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1486.313700] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.318035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.333076] Lustre: Skipped 1 previous similar message [ 1489.465268] Lustre: server umount lustre-OST0000 complete [ 1495.521468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1495.527679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.535159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1495.539168] Lustre: Skipped 1 previous similar message [ 1496.548650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1496.552143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.664614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.669519] Lustre: Skipped 2 previous similar messages [ 1506.271154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1506.408647] Lustre: server umount lustre-MDT0000 complete [ 1508.651066] LustreError: 69117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486403 with bad export cookie 6450146057470387557 [ 1508.651585] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1508.657589] LustreError: 69117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1508.865416] Lustre: server umount lustre-MDT0001 complete [ 1514.082904] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 11:06:47 (1782486407) [ 1520.287412] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1542.934484] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1548.694921] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1569.583459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.772693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1569.801988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1569.857782] Lustre: lustre-MDT0000: new disk, initializing [ 1569.911224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.923434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1572.473619] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1579.160738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1579.224918] Lustre: 76299:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1579.265922] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1579.269320] Lustre: Skipped 1 previous similar message [ 1579.316298] Lustre: lustre-MDT0001: new disk, initializing [ 1579.367706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1579.393755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1579.402871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1581.650955] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1587.466905] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1590.693848] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1595.278932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1595.456053] Lustre: lustre-OST0000: new disk, initializing [ 1595.464647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1595.469770] Lustre: 78267:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1595.531320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.321892] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1604.107584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1604.112770] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1604.177849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1605.424883] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1607.991769] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1608.126820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1610.742153] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1610.887995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1614.307276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1614.307326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.315170] LustreError: Skipped 1 previous similar message [ 1614.317135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1614.325647] Lustre: Skipped 1 previous similar message [ 1618.110942] Lustre: server umount lustre-OST0000 complete [ 1624.548198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.549363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1624.563626] Lustre: Skipped 1 previous similar message [ 1624.564345] Lustre: Skipped 1 previous similar message [ 1626.835572] Lustre: server umount lustre-MDT0000 complete [ 1629.154326] LustreError: 76291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486523 with bad export cookie 6450146057470388439 [ 1629.158245] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1629.161101] LustreError: 76291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1629.378588] Lustre: server umount lustre-MDT0001 complete [ 1634.218746] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 11:08:47 (1782486527) [ 1637.821850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.067210] LustreError: 80675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.081295] LustreError: 80675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1638.113339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.345248] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1643.491085] LustreError: 80676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.794983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.203325] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1650.930291] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1653.690839] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1657.504957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.963545] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1664.929472] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1666.835628] Lustre: 83342:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting f2682312-7117-4d92-8e55-39464ffbba04 at adminstrative request [ 1668.069144] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1668.074699] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1678.240933] Lustre: MGS: Client f2682312-7117-4d92-8e55-39464ffbba04 (at 0@lo) reconnecting [ 1678.245738] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x598385aa38e8ac49 to 0x598385aa38e8aed4 [ 1678.251968] Lustre: MGC192.168.204.154@tcp: Connection restored to 0@lo (at 0@lo) [ 1700.735429] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1700.833171] Lustre: lustre-OST0001: new disk, initializing [ 1700.835887] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1700.839700] Lustre: 83738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1700.882658] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1700.885745] Lustre: Skipped 2 previous similar messages [ 1704.158923] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1708.569588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1708.584869] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1708.631849] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1709.810816] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1713.636860] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1713.639148] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1713.643731] LustreError: Skipped 1 previous similar message [ 1713.650761] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1713.656952] Lustre: Skipped 1 previous similar message [ 1713.665968] Lustre: Skipped 2 previous similar messages [ 1717.575640] Lustre: server umount lustre-OST0001 complete [ 1718.762516] LustreError: 82492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.778626] LustreError: 82492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1721.312235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1723.878532] LustreError: 82492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1723.886854] LustreError: 82492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1729.004026] LustreError: 82494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.013919] LustreError: 82494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1734.111114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1734.223346] Lustre: server umount lustre-OST0000 complete [ 1737.184641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1737.194372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1737.201614] Lustre: Skipped 2 previous similar messages [ 1742.562595] Lustre: server umount lustre-MDT0000 complete [ 1744.352081] LustreError: 80676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1744.365080] LustreError: 80676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1744.739611] LustreError: 80656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486639 with bad export cookie 6450146057470389972 [ 1744.744356] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.749317] LustreError: 80656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1744.943850] Lustre: server umount lustre-MDT0001 complete [ 1749.583444] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 11:10:42 (1782486642) [ 1750.601389] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1751.641649] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 11:10:44 (1782486644) [ 1752.866407] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1754.336589] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 11:10:47 (1782486647) [ 1755.671940] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1756.812451] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 11:10:49 (1782486649) [ 1758.208534] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 1760.227915] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 11:10:52 (1782486652) [ 1763.013088] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1769.728909] loop: module loaded [ 1769.746902] loop0: detected capacity change from 0 to 4294967296 [ 1769.796857] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1772.893618] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1776.773901] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1782.060342] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1820.480091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1820.594511] LustreError: 87864:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1820.623225] LustreError: 87864:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1820.643371] LustreError: 87864:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1820.660920] LustreError: 87864:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1850.208828] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 1859.536387] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 1887.587210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.845944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1887.876253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1887.933347] Lustre: lustre-MDT0000: new disk, initializing [ 1888.004354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.023488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1891.193245] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1899.948992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.042171] Lustre: 90731:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1900.296893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1903.719577] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1911.056841] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1916.135957] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1921.630673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1921.900626] Lustre: 92701:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1927.385305] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1929.262432] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1929.373537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1934.948467] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1938.899650] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1939.078874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1942.711665] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1942.864710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1944.548060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1944.553886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.572343] Lustre: Skipped 1 previous similar message [ 1944.580951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1944.585641] Lustre: Skipped 10 previous similar messages [ 1950.711561] Lustre: server umount lustre-OST0000 complete [ 1968.095391] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1968.236786] Lustre: server umount lustre-MDT0000 complete [ 1970.658289] LustreError: 90742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1970.673860] LustreError: 90742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1971.438745] LustreError: 90724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486866 with bad export cookie 6450146057470390595 [ 1971.448323] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1971.450707] LustreError: 90724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1971.739206] Lustre: server umount lustre-MDT0001 complete [ 1978.308690] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 11:14:31 (1782486871) [ 1983.940347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.081981] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 1995.231588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.033762] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2004.844740] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2009.282553] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2015.516541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2021.486448] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2028.057993] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2032.497503] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2032.699664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2036.795593] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2036.965237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2043.036805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2043.169515] Lustre: lustre-OST0001: new disk, initializing [ 2043.171336] Lustre: Skipped 2 previous similar messages [ 2043.174321] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2043.178698] Lustre: Skipped 3 previous similar messages [ 2043.183199] Lustre: 98481:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2043.268517] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.277359] Lustre: Skipped 5 previous similar messages [ 2048.885304] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2052.662929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2052.670572] Lustre: Skipped 2 previous similar messages [ 2052.679177] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2052.759469] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2056.919228] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2061.051699] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2061.226540] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2065.519502] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2065.731787] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2083.081338] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2107.610987] Lustre: Failing over lustre-MDT0000 [ 2108.051322] Lustre: server umount lustre-MDT0000 complete [ 2108.384328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2108.392636] LustreError: Skipped 1 previous similar message [ 2108.395839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.413614] Lustre: Skipped 3 previous similar messages [ 2108.420874] LustreError: 95111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.437870] LustreError: 95111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2125.280103] Lustre: 66548:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782487003/real 1782487003] req@000000002bc5844f x1869072359994496/t0(0) o400->MGC192.168.204.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782487019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.312298] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2130.147584] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2130.150237] LDISKFS-fs (dm-0): recovery complete [ 2130.158761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.499520] LustreError: 66547:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007924c69d x1869072360002816/t0(0) o250->MGC192.168.204.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2134.877042] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2135.763554] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2135.770527] Lustre: 95107:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2138.825275] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2140.137352] Lustre: 95110:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2140.152023] Lustre: 95110:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 1 previous similar message [ 2140.166166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2140.244323] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2140.573794] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2140.911654] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2140.924326] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 1 previous similar message [ 2141.255670] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2141.266306] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 1 previous similar message [ 2141.276457] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2141.300561] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 7 previous similar messages [ 2141.940530] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2141.948237] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 2 previous similar messages [ 2142.271986] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2142.280912] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 2 previous similar messages [ 2143.568427] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2143.580532] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 6 previous similar messages [ 2144.234343] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2144.244247] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 6 previous similar messages [ 2144.575163] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2144.583499] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 6 previous similar messages [ 2147.768219] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2147.781127] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 12 previous similar messages [ 2148.408270] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2148.426686] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 12 previous similar messages [ 2148.743116] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2148.747529] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 12 previous similar messages [ 2156.050581] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2156.060428] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 25 previous similar messages [ 2156.712260] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2156.723799] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 25 previous similar messages [ 2157.039269] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2157.050039] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 25 previous similar messages [ 2172.292538] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2172.301906] Lustre: 101326:0:(ldlm_lib.c:2071:extend_recovery_timer()) Skipped 50 previous similar messages [ 2172.942952] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2172.953267] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 50 previous similar messages [ 2173.271161] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2173.280271] LustreError: 101326:0:(ldlm_lib.c:2684:replay_request_or_update()) Skipped 50 previous similar messages [ 2195.071216] Lustre: 101326:0:(ldlm_lib.c:2380:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2195.085375] Lustre: 101326:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2195.094843] Lustre: 101326:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 153c4285-35b6-402e-873a-241d92b61f40@192.168.204.54@tcp [ 2195.107738] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2195.113494] Lustre: 101326:0:(ldlm_lib.c:1900:abort_req_replay_queue()) @@@ aborted: req@00000000a0569ba4 x1869072348284672/t0(8589934769) o101->153c4285-35b6-402e-873a-241d92b61f40@192.168.204.54@tcp:50/0 lens 592/0 e 0 to 0 dl 1782487100 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2195.155674] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2195.163479] Lustre: lustre-MDT0000: Denying connection for new client 153c4285-35b6-402e-873a-241d92b61f40 (at 192.168.204.54@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2195.183922] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2195.214792] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2195.223281] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2195.233460] Lustre: Skipped 2 previous similar messages [ 2195.251721] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2195.261518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:77 to 0x280000401:577) [ 2210.274093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2210.281950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2210.285688] Lustre: Skipped 10 previous similar messages [ 2214.627612] Lustre: server umount lustre-OST0000 complete [ 2216.928928] LustreError: 96931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.948172] LustreError: 96931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 2231.263461] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2231.368265] Lustre: server umount lustre-OST0001 complete [ 2237.410229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2237.416355] Lustre: Skipped 8 previous similar messages [ 2239.344732] Lustre: server umount lustre-MDT0000 complete [ 2241.334249] LustreError: 95091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487135 with bad export cookie 6450146057470441807 [ 2241.335208] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2241.341981] LustreError: 95091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2241.540187] Lustre: server umount lustre-MDT0001 complete [ 2245.686770] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 11:18:59 (1782487139) [ 2249.553983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.558319] Lustre: *** cfs_fail_loc=197, val=0*** [ 2265.055938] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2270.175497] LustreError: 103300:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 2275.310729] Lustre: *** cfs_fail_loc=197, val=0*** [ 2275.312939] Lustre: Skipped 65 previous similar messages [ 2298.885536] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2302.756403] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2303.800370] Lustre: server umount lustre-OST0000 complete [ 2307.574880] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 11:20:01 (1782487201) [ 2322.444925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.691646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.694860] Lustre: Skipped 2 previous similar messages [ 2324.590355] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2329.817425] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2332.207828] LustreError: 104950:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2332.213483] LustreError: 104950:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2332.216736] LustreError: 104950:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2332.357296] Lustre: server umount lustre-MDT0000 complete [ 2346.445246] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2350.515203] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2365.422053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.538665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2365.555367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2365.595612] Lustre: lustre-MDT0000: new disk, initializing [ 2365.639450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2367.178754] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2371.933212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.978189] Lustre: 109117:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2372.103403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2373.759137] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2378.237997] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2380.351266] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2383.126177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2383.276327] Lustre: 111086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2385.544786] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2389.057700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2389.974507] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2391.849159] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2391.957582] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2393.713767] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2393.818749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2399.200463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2399.205033] LustreError: Skipped 5 previous similar messages [ 2399.208107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.211526] Lustre: Skipped 12 previous similar messages [ 2400.669680] Lustre: server umount lustre-OST0000 complete [ 2408.441606] Lustre: server umount lustre-MDT0000 complete [ 2409.440137] LustreError: 109929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.448061] LustreError: 109929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2410.252752] LustreError: 109912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487304 with bad export cookie 6450146057470452384 [ 2410.255758] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2410.258271] LustreError: 109912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2410.407739] Lustre: server umount lustre-MDT0001 complete [ 2414.164363] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 11:21:47 (1782487307) [ 2417.850203] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2419.122605] Key type lgssc unregistered [ 2419.268547] LNet: 113570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.273513] LNetError: 113570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.280851] LNet: Removed LNI 192.168.204.154@tcp [ 2419.643176] Key type .llcrypt unregistered [ 2419.644484] Key type ._llcrypt unregistered [ 2427.158574] Key type ._llcrypt registered [ 2427.160436] Key type .llcrypt registered [ 2427.207995] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2435.014625] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2435.425894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2435.498485] alg: No test for adler32 (adler32-zlib) [ 2436.396418] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2436.497910] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2438.095216] Key type lgssc registered [ 2438.581682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2457.145969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2457.156909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.290451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2458.309900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2458.352262] Lustre: lustre-MDT0000: new disk, initializing [ 2458.412863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.425162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2460.203955] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2464.802304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2464.921461] Lustre: lustre-OST0000: new disk, initializing [ 2464.924182] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2464.928079] Lustre: Skipped 1 previous similar message [ 2464.933340] Lustre: 117719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2464.967606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.244626] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2471.927088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2471.991908] Lustre: lustre-OST0001: new disk, initializing [ 2471.994870] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2471.998749] Lustre: 118575:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2472.035804] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2473.768890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2473.774806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2473.798502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2473.804462] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2474.458391] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2479.072960] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.075600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2479.078310] Lustre: Skipped 1 previous similar message [ 2479.087039] Lustre: Skipped 1 previous similar message [ 2484.193449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2484.196830] Lustre: Skipped 1 previous similar message [ 2485.139731] Lustre: server umount lustre-MDT0000 complete [ 2487.841593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2489.864230] LustreError: 116903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487384 with bad export cookie 11332370767325366962 [ 2489.866800] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2489.871255] LustreError: 116903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2489.917805] Lustre: server umount lustre-OST0000 complete [ 2491.650240] Lustre: server umount lustre-OST0001 complete [ 2495.141090] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 11:23:08 (1782487388) [ 2499.006662] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2509.269145] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 11:23:22 (1782487402) [ 2525.585233] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 2529.716342] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2544.446599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.560978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2544.577698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2544.623829] Lustre: lustre-MDT0000: new disk, initializing [ 2544.664166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2544.673329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2544.676891] Lustre: Skipped 1 previous similar message [ 2546.559681] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2552.022374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.068044] Lustre: 125313:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2552.096562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2552.099574] Lustre: Skipped 1 previous similar message [ 2552.140244] Lustre: lustre-MDT0001: new disk, initializing [ 2552.185399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2552.204314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2552.210574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2552.214823] Lustre: Skipped 1 previous similar message [ 2554.110581] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2558.810980] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2561.052904] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2564.121533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2564.284404] Lustre: lustre-OST0000: new disk, initializing [ 2564.287024] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2564.290709] Lustre: 127279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2564.338048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.928515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2564.933555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2564.978044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2566.873085] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2571.921664] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2574.104506] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2574.196154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2576.214416] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2576.312288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2580.451958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2580.452395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.466941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2580.469922] Lustre: Skipped 1 previous similar message [ 2583.404856] Lustre: server umount lustre-OST0000 complete [ 2585.568879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2585.570545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.577022] Lustre: Skipped 2 previous similar messages [ 2585.586121] Lustre: Skipped 1 previous similar message [ 2590.689350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2590.695722] Lustre: Skipped 1 previous similar message [ 2599.391220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2599.488564] Lustre: server umount lustre-MDT0000 complete [ 2601.374382] LustreError: 127278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487495 with bad export cookie 11332370767325368131 [ 2601.375539] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2601.382687] LustreError: 127278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2601.523974] Lustre: server umount lustre-MDT0001 complete [ 2605.244301] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 11:24:58 (1782487498) [ 2608.178449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.406486] LustreError: 129687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2608.419131] LustreError: 129687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2608.445827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.177682] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2613.650228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2613.728850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2613.732044] Lustre: Skipped 2 previous similar messages [ 2615.722748] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2619.187430] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2621.796344] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2624.951820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2627.511772] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2630.727776] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2642.119862] Lustre: *** cfs_fail_loc=159, val=0*** [ 2642.123377] Lustre: Skipped 1 previous similar message [ 2658.534256] Lustre: lustre-MDT0000: Client 663df0b0-6286-4235-af26-bf8c20ee9cc2 (at 192.168.204.54@tcp) reconnecting [ 2659.157539] Lustre: *** cfs_fail_loc=159, val=0*** [ 2659.159783] Lustre: Skipped 5 previous similar messages [ 2675.428630] Lustre: lustre-MDT0000: Client 663df0b0-6286-4235-af26-bf8c20ee9cc2 (at 192.168.204.54@tcp) reconnecting [ 2692.063867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2692.067338] LustreError: Skipped 1 previous similar message [ 2692.069213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.075578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2695.138770] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2696.536573] Lustre: server umount lustre-OST0000 complete [ 2700.256559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.262832] Lustre: Skipped 2 previous similar messages [ 2704.230337] Lustre: server umount lustre-MDT0000 complete [ 2705.377130] LustreError: 130925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.383710] LustreError: 130925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2705.859508] LustreError: 129669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487600 with bad export cookie 11332370767325369013 [ 2705.861626] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.866543] LustreError: 129669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2706.004924] Lustre: server umount lustre-MDT0001 complete [ 2711.154987] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2712.783288] Key type lgssc unregistered [ 2712.969446] LNet: 133485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2712.975679] LNetError: 133485:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2712.993571] LNet: Removed LNI 192.168.204.154@tcp [ 2713.469188] Key type .llcrypt unregistered [ 2713.473230] Key type ._llcrypt unregistered [ 2727.885835] Key type ._llcrypt registered [ 2727.887287] Key type .llcrypt registered [ 2727.942427] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2728.532070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2728.553197] alg: No test for adler32 (adler32-zlib) [ 2729.422974] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2729.526658] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2731.127877] Key type lgssc registered [ 2731.583231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2735.024110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2735.036409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.278299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.998468] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2741.927661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.086603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2743.767925] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2746.927420] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2749.635581] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2752.662314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.820154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.048348] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2758.024582] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2760.998890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 2773.344437] Lustre: *** cfs_fail_loc=159, val=0*** [ 2789.601910] Lustre: lustre-MDT0000: Client 18bdf719-d79a-4228-85e5-1dd2defb96ba (at 192.168.204.54@tcp) reconnecting [ 2790.592449] Lustre: *** cfs_fail_loc=159, val=0*** [ 2790.597502] Lustre: Skipped 1 previous similar message [ 2805.982815] Lustre: lustre-MDT0001: Client 18bdf719-d79a-4228-85e5-1dd2defb96ba (at 192.168.204.54@tcp) reconnecting [ 2806.595168] Lustre: *** cfs_fail_loc=159, val=0*** [ 2806.597669] Lustre: Skipped 2 previous similar messages [ 2822.887313] Lustre: lustre-MDT0001: Client 18bdf719-d79a-4228-85e5-1dd2defb96ba (at 192.168.204.54@tcp) reconnecting [ 2824.542304] Lustre: 137930:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2825.680454] Lustre: *** cfs_fail_loc=159, val=0*** [ 2825.683074] Lustre: Skipped 4 previous similar messages [ 2841.322466] Lustre: lustre-MDT0000: Client ffaa0357-1c9b-436e-b0ea-68341e0f9529 (at 192.168.204.54@tcp) reconnecting [ 2842.053646] Lustre: *** cfs_fail_loc=159, val=0*** [ 2842.055733] Lustre: Skipped 14 previous similar messages [ 2858.236693] Lustre: lustre-MDT0000: Client ffaa0357-1c9b-436e-b0ea-68341e0f9529 (at 192.168.204.54@tcp) reconnecting [ 2860.163474] Lustre: 138222:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 2878.945327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2878.951139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.964743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2879.972603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.975054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2880.039322] Lustre: Skipped 1 previous similar message [ 2884.260723] Lustre: server umount lustre-OST0000 complete [ 2895.337686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.363858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.397742] Lustre: Skipped 1 previous similar message [ 2898.379889] Lustre: server umount lustre-MDT0000 complete [ 2899.564547] LustreError: 134873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2899.618222] LustreError: 134873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2905.060350] LustreError: 136118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2905.072542] LustreError: 136118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2906.121493] LustreError: 134857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487800 with bad export cookie 6140643673246558914 [ 2906.130571] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2906.152232] LustreError: 134857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2907.409507] Lustre: server umount lustre-MDT0001 complete [ 2926.905665] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 2932.551072] Key type lgssc unregistered [ 2933.136711] LNet: 139326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.155445] LNetError: 139326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2933.206063] LNet: Removed LNI 192.168.204.154@tcp [ 2934.457856] Key type .llcrypt unregistered [ 2934.459474] Key type ._llcrypt unregistered [ 2968.220265] Key type ._llcrypt registered [ 2968.231026] Key type .llcrypt registered [ 2968.468184] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 2970.539833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2970.586234] alg: No test for adler32 (adler32-zlib) [ 2971.999963] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2972.505169] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 2974.297540] Key type lgssc registered [ 2976.417708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2985.183842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2985.196287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.546691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.026089] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 2996.720876] LustreError: 140741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2996.743267] LustreError: 140741:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2999.255949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.937781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3004.699275] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3011.666731] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3017.607833] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3024.115189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3024.457751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.527437] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3030.820374] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3031.655514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3039.228473] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3045.179948] Lustre: 143330:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3048.610093] Lustre: 143379:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3050.168258] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3051.280948] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3055.897300] Lustre: 143528:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3056.892798] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3058.124720] Lustre: 143636:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3066.852693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3066.873882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.903848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3067.875282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3067.876815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3067.912988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3069.579560] Lustre: server umount lustre-OST0000 complete [ 3076.581109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3076.596276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.608781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.613783] Lustre: Skipped 1 previous similar message [ 3083.235763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3083.246741] Lustre: Skipped 3 previous similar messages [ 3088.354290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3088.358993] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 3088.366524] Lustre: Skipped 1 previous similar message [ 3088.805621] Lustre: server umount lustre-MDT0000 complete [ 3092.450373] LustreError: 140742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.462802] LustreError: 140742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3092.577304] LustreError: 143428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487987 with bad export cookie 1973399116718075322 [ 3092.584040] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.588882] LustreError: 143428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3092.995028] Lustre: server umount lustre-MDT0001 complete [ 3103.353228] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3106.177896] Key type lgssc unregistered [ 3106.528333] LNet: 144689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3106.539258] LNetError: 144689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3107.569359] LNet: Removed LNI 192.168.204.154@tcp [ 3108.492615] Key type .llcrypt unregistered [ 3108.495087] Key type ._llcrypt unregistered [ 3132.593816] Key type ._llcrypt registered [ 3132.596208] Key type .llcrypt registered [ 3132.734257] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3134.006791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3134.028635] alg: No test for adler32 (adler32-zlib) [ 3135.126854] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 3135.489784] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3137.210144] Key type lgssc registered [ 3138.502940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3147.195238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3147.226902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.780485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3153.808846] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3159.011203] LustreError: 146105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.031562] LustreError: 146105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3163.052796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.361831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3168.195752] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3176.592988] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3181.607432] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3188.634105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.139973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.315935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 3196.458936] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3204.532185] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3207.611243] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 3214.635396] Lustre: *** cfs_fail_loc=159, val=0*** [ 3231.034221] Lustre: lustre-MDT0000: Client 7d65d26b-d047-47ee-99f4-5c75f7998139 (at 192.168.204.54@tcp) reconnecting [ 3236.848629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3236.851513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3236.861244] LustreError: Skipped 2 previous similar messages [ 3236.871668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3238.358973] Lustre: server umount lustre-OST0000 complete [ 3243.488793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3243.505934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3243.511017] Lustre: Skipped 1 previous similar message [ 3243.534122] Lustre: Skipped 1 previous similar message [ 3248.620044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.631186] Lustre: Skipped 1 previous similar message [ 3253.730794] Lustre: 145417:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782488132/real 1782488132] req@00000000cb229ef3 x1869074168582016/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782488148 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3253.733115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3253.764110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.776927] Lustre: Skipped 1 previous similar message [ 3256.287794] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3256.535575] Lustre: server umount lustre-MDT0000 complete [ 3258.850074] LustreError: 146106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.873520] LustreError: 146106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3259.872103] Lustre: 145416:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782488138/real 1782488138] req@000000005331011b x1869074168583552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782488154 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3259.924285] Lustre: 145416:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3259.985043] LustreError: 147957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488154 with bad export cookie 16341144443412349629 [ 3259.996028] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3260.012449] LustreError: 147957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3260.478387] Lustre: server umount lustre-MDT0001 complete [ 3272.364401] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3275.697210] Key type lgssc unregistered [ 3276.082834] LNet: 149770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3276.093072] LNetError: 149770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3276.120996] LNet: Removed LNI 192.168.204.154@tcp [ 3276.927728] Key type .llcrypt unregistered [ 3276.932107] Key type ._llcrypt unregistered [ 3305.235243] Key type ._llcrypt registered [ 3305.238291] Key type .llcrypt registered [ 3305.429426] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3307.110356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3307.175872] alg: No test for adler32 (adler32-zlib) [ 3308.386526] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 3308.706274] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3310.391324] Key type lgssc registered [ 3311.677494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3320.391198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3320.407950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.087686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3327.325456] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3331.567622] LustreError: 151184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.582974] LustreError: 151184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3336.677594] LustreError: 151183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3337.350823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.807513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3342.792199] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3353.102116] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3360.855698] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3368.882724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.397699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.350952] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3382.722558] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 3383.795759] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3391.938950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 3416.717591] Lustre: lustre-MDT0001: evicting 7f72ed65-6f6d-40f1-9e65-728ea77d6ac7 (at 192.168.204.54@tcp) by administrative request [ 3416.745495] Lustre: Skipped 1 previous similar message [ 3449.316503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.335209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.366979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3453.421628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3453.422491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.471826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3454.744779] Lustre: server umount lustre-OST0000 complete [ 3460.577955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3460.592096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.606390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.622210] Lustre: Skipped 1 previous similar message [ 3463.649322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.651714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3463.704713] Lustre: Skipped 1 previous similar message [ 3465.731974] Lustre: server umount lustre-MDT0000 complete [ 3468.769916] LustreError: 151183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3468.804757] LustreError: 151183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3469.819689] LustreError: 151165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488364 with bad export cookie 13988566203144831179 [ 3469.822717] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3469.831419] LustreError: 151165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3470.496457] Lustre: server umount lustre-MDT0001 complete [ 3486.296440] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3488.876587] Key type lgssc unregistered [ 3489.176910] LNet: 155285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3489.183361] LNetError: 155285:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3489.213659] LNet: Removed LNI 192.168.204.154@tcp [ 3490.084424] Key type .llcrypt unregistered [ 3490.090445] Key type ._llcrypt unregistered [ 3579.555710] Key type ._llcrypt registered [ 3579.563748] Key type .llcrypt registered [ 3579.732295] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3580.727878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3580.760736] alg: No test for adler32 (adler32-zlib) [ 3582.041188] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 3582.354733] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3584.236089] Key type lgssc registered [ 3585.680392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3592.671914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3592.691381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.467239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.941518] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3604.961775] LustreError: 157774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3604.972651] LustreError: 157774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3605.832098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.161408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3609.577458] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3615.010638] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3618.065714] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3622.271357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3622.475933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.699372] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3629.831885] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3630.703824] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 3639.921076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 3646.432229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3646.437598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.456156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.532113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3650.533619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.540785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.559658] Lustre: Skipped 1 previous similar message [ 3652.498603] Lustre: server umount lustre-OST0000 complete [ 3655.648742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.652878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.656247] Lustre: Skipped 1 previous similar message [ 3655.662710] Lustre: Skipped 1 previous similar message [ 3660.624679] Lustre: server umount lustre-MDT0000 complete [ 3660.768811] LustreError: 158877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3660.778013] LustreError: 158877:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3662.792584] LustreError: 157755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488557 with bad export cookie 15555919342939689096 [ 3662.794663] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3662.800469] LustreError: 157755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3662.984254] Lustre: server umount lustre-MDT0001 complete [ 3668.145810] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3669.537679] Key type lgssc unregistered [ 3669.729633] LNet: 161334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3669.736323] LNetError: 161334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3669.749551] LNet: Removed LNI 192.168.204.154@tcp [ 3670.199395] Key type .llcrypt unregistered [ 3670.201900] Key type ._llcrypt unregistered [ 3683.370659] Key type ._llcrypt registered [ 3683.372280] Key type .llcrypt registered [ 3683.466726] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3684.166896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3684.177678] alg: No test for adler32 (adler32-zlib) [ 3685.067279] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 3685.171378] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3686.759158] Key type lgssc registered [ 3687.156920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3692.559041] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 11:43:06 (1782488586) [ 3698.510237] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 3702.101767] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3714.541743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3714.548698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3715.645638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3715.656957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3715.692740] Lustre: lustre-MDT0000: new disk, initializing [ 3715.724585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.732275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3717.164285] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3721.436031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3721.477290] Lustre: 165565:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3721.500385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3721.502823] Lustre: Skipped 1 previous similar message [ 3721.539464] Lustre: lustre-MDT0001: new disk, initializing [ 3721.571076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3721.584611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3721.588280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3723.007710] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3726.809369] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3728.670214] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3730.997138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3731.132961] Lustre: lustre-OST0000: new disk, initializing [ 3731.135620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3731.139198] Lustre: 167538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3731.175937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3731.673616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3731.678731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3731.721683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3733.240930] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3737.090161] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3742.177652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3742.181128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3742.182414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3742.186234] Lustre: Skipped 1 previous similar message [ 3746.516550] Lustre: server umount lustre-OST0000 complete [ 3752.417033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3752.417830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.418787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3752.418792] Lustre: Skipped 1 previous similar message [ 3752.420053] LustreError: Skipped 1 previous similar message [ 3752.426177] Lustre: Skipped 1 previous similar message [ 3754.084817] Lustre: server umount lustre-MDT0000 complete [ 3755.487308] LustreError: 166366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488650 with bad export cookie 2195214935599433448 [ 3755.491812] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3755.493958] LustreError: 166366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3755.640969] Lustre: server umount lustre-MDT0001 complete [ 3759.631447] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 3760.850986] Key type lgssc unregistered [ 3760.983480] LNet: 169443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3760.986674] LNetError: 169443:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3760.994360] LNet: Removed LNI 192.168.204.154@tcp [ 3761.325188] Key type .llcrypt unregistered [ 3761.326800] Key type ._llcrypt unregistered [ 3770.954581] Key type ._llcrypt registered [ 3770.956290] Key type .llcrypt registered [ 3770.994210] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 3771.374389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3771.437093] alg: No test for adler32 (adler32-zlib) [ 3772.304125] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 3772.398110] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 3773.991224] Key type lgssc registered [ 3774.508720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3777.558494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3777.565813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.723186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3780.180065] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3783.332519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3783.487501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3784.998727] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3787.596195] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3789.590252] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3791.885909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3791.995870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3793.968690] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 3796.454235] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3804.328084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3809.373426] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3809.382698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.388601] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3809.391933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3809.397249] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3809.749765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3810.098516] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3810.102367] Lustre: Skipped 1 previous similar message [ 3810.109428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.115264] Lustre: Skipped 1 previous similar message [ 3810.117516] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3810.119623] Lustre: Skipped 1 previous similar message [ 3810.120864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3810.125976] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3810.129104] Lustre: Skipped 1 previous similar message [ 3810.433761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3810.764384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3811.085831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3811.418124] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3811.421752] Lustre: Skipped 3 previous similar messages [ 3811.430052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.435673] Lustre: Skipped 3 previous similar messages [ 3811.437900] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3811.440726] Lustre: Skipped 3 previous similar messages [ 3811.442940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3811.447933] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3811.451696] Lustre: Skipped 3 previous similar messages [ 3811.778898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3812.112411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3812.458850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3812.858133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3813.207681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3813.528316] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3813.530588] Lustre: Skipped 5 previous similar messages [ 3813.536695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.542419] Lustre: Skipped 5 previous similar messages [ 3813.544781] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3813.547889] Lustre: Skipped 5 previous similar messages [ 3813.550213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3813.554457] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3813.556626] Lustre: Skipped 5 previous similar messages [ 3813.858941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3814.199684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3814.512745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3814.833931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3815.147622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3815.476193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3815.801226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3816.129850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3816.444302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3816.798068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3817.138733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3817.508681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3817.879261] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3817.882330] Lustre: Skipped 12 previous similar messages [ 3817.889437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3817.894285] Lustre: Skipped 12 previous similar messages [ 3817.896026] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3817.898707] Lustre: Skipped 12 previous similar messages [ 3817.900272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3817.904502] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3817.907268] Lustre: Skipped 12 previous similar messages [ 3818.239041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3818.574234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3818.918704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3819.253368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3819.589699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3819.921264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3820.278569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3820.620130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3820.944189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3821.273593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3821.607343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3821.937088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3822.282074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3822.634190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3822.992806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3823.290594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3823.608650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3823.929407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3824.250523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3824.545876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3824.848816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3825.142488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3825.446550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3825.751833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3826.059406] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3826.061478] Lustre: Skipped 24 previous similar messages [ 3826.067763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.072455] Lustre: Skipped 24 previous similar messages [ 3826.074697] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3826.077050] Lustre: Skipped 24 previous similar messages [ 3826.078613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3826.083214] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3826.085706] Lustre: Skipped 24 previous similar messages [ 3826.407970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3826.738461] LustreError: 170854:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 3826.742639] LustreError: 170854:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 3826.743227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3827.065150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3827.384643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3827.701992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3828.043316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3828.374723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3828.676792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3828.982728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3829.344759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3829.687746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3830.016936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3830.369528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3830.680137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3830.995337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3831.289483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3831.599213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3831.901233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3832.208868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3832.536147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3832.852306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3833.160052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3833.476803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3833.780863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3834.111707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3834.411932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3834.708508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3835.022246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3835.340708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3835.659595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3835.965332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3836.274750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3836.595066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3836.881033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3837.168717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3837.456150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3837.791900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3838.099525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3838.394096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3838.685543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3838.997131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3839.294504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3839.603026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3839.921725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3840.263251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3840.590423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3840.918500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3841.233931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3841.583217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3841.911905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3842.232147] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3842.234953] Lustre: Skipped 50 previous similar messages [ 3842.240736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.244505] Lustre: Skipped 50 previous similar messages [ 3842.246248] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3842.249228] Lustre: Skipped 50 previous similar messages [ 3842.251138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3842.254882] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3842.257707] Lustre: Skipped 50 previous similar messages [ 3842.566469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3842.895826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3843.221551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3843.550962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3843.907768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3844.296925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3844.692456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3845.460242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3845.863544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3846.229114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3846.614601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3846.958643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3847.340563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3847.737308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3848.153694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3848.558252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3849.035599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3849.544154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3850.073343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3850.648340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3851.181602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3851.812258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3852.673169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3853.356728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3854.071232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3854.688025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3855.294392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3855.988928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3856.675608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3857.303795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3858.018495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3858.617850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3859.457030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3860.080983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3861.311810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3861.976295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3862.564826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3863.128509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3864.035785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3864.535076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3865.671129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3866.280110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3866.962528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3867.588836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3868.231652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3868.921647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3869.530822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3870.196585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3871.293649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3872.078714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3872.779968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3873.529855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3874.269747] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3874.283114] Lustre: Skipped 52 previous similar messages [ 3874.300468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3874.309675] Lustre: Skipped 52 previous similar messages [ 3874.313308] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3874.321321] Lustre: Skipped 52 previous similar messages [ 3874.328489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3874.339817] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3874.353951] Lustre: Skipped 52 previous similar messages [ 3875.080097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3875.794887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3876.544508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3877.239977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3877.846579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3878.563539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3879.193025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3879.960021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3880.637833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3881.250355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3881.834566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3882.607637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3883.237893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3884.086875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3884.759057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3885.474244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3886.231837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3886.979182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3887.653560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3888.367876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3889.515101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3890.168254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3890.923729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3891.664218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3892.496160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3893.478513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3895.062372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3895.741726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3896.473283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3897.508415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3898.164799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3898.972653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3899.515581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3900.301165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3900.861716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3901.611818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3902.232659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3902.877212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3903.679097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3904.450883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3905.176156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3905.971768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3907.015647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3908.166161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3909.150953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3910.085410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3910.787705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3911.662460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3912.389382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3913.315882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3914.207929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3915.037305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3915.784392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3916.555438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3917.256588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3917.905982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3918.634562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3919.515993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3920.222713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3920.839661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3921.497464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3922.239190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3922.983852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3923.742314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3924.505251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3925.515577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3926.396911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3927.147470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3927.967755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3928.874842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3929.664805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3930.473890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3931.233142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3932.074307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3932.784194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3933.576933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3934.205984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3934.988827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3935.788867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3936.619558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3937.347309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3938.069871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3938.774361] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3938.776821] Lustre: Skipped 82 previous similar messages [ 3938.795643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.803547] Lustre: Skipped 82 previous similar messages [ 3938.808379] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3938.812932] Lustre: Skipped 82 previous similar messages [ 3938.815404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3938.820965] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3938.827154] Lustre: Skipped 82 previous similar messages [ 3939.379596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3940.118270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3940.938944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3941.496669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3942.171492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3942.767563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3943.339253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3943.882647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3944.421625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3945.088601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3945.706457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3946.183590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3946.721201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3947.202650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3947.645374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3948.143586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3948.576860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3949.104673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3949.672414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3950.281330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3950.864244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3951.489462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3952.223852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3952.869938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3953.419825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3953.995083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3954.636268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3955.204068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3955.773263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3956.465379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3957.043275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3957.650108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3958.241059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3958.721550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3959.175229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3959.592300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3960.073630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3960.597983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3961.305160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3961.937267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3962.556334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3963.153550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3963.844286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3964.443932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3965.174377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3965.898646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3966.540438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3967.164287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3967.769196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3973.317879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3973.870732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3974.418732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3975.102576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3975.638660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3976.132859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3976.737070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3977.324743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3977.922805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3978.533534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3979.168350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3979.840253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3980.424759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3981.065020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3981.621537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3982.178942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3982.730253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3983.291325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3983.940808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3984.520524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3985.170345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3985.867483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3986.538059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3987.180814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3987.873284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3988.549809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3989.239883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3989.828989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3990.500602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3991.106851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3991.783684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3992.402545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3993.014712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3993.640964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3994.179626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3994.753645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3995.257246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3995.794747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3996.267355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3996.755720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3997.277797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3997.837425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3998.350969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3998.870547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3999.384282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3999.892840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4000.410039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4000.872294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4001.348762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4001.873435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4002.394676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4002.932639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4003.466747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4004.076660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4004.560538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4005.023438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4005.477750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4005.958625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4006.482785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4007.013674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4007.533874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4008.045552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4008.657566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4009.243494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4009.725397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4010.202238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4010.715775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4011.219928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4011.812106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4012.402901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4013.000243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4013.549550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4014.177223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4014.702131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4015.201530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4015.703347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4016.198856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4016.762452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4017.240684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4017.730408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4018.241644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4018.772446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4019.272419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4019.756585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4020.279293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4020.777290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4021.276280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4021.815636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4022.268085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4022.828179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4023.330177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4023.754099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4024.153170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4024.558891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4024.970587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4025.419561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4025.906555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4026.388559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4026.900769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4027.425966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4027.960975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4028.443203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4028.890875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4029.305785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4029.742459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4030.233927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4030.693325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4031.191356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4031.671449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4032.176648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4032.801904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4033.417807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4033.997322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4034.645678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4035.372734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4036.032519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4036.735489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4037.326499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4037.994132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4038.585528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4039.182721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4039.860866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4040.568803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4041.220536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4041.779724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4042.365647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4042.952774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4043.496577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4044.090303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4044.577940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4045.073931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4045.633598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4046.194173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4046.808082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4047.406758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4048.007794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4048.547786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4049.046333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4049.542794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4050.057592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4050.563418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4051.032857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4051.496506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4052.058543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4052.601331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4053.101939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4053.626871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4054.154280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4054.637235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4055.137816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4055.641155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4056.229952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4056.781680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4057.350585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4057.883880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4058.398553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4058.977107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4059.514416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4060.039476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4060.558862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4061.097301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4061.614853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4062.122496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4062.674478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4063.178696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4063.658899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4064.210760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4064.727068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4065.210602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4065.745922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4066.290384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4066.833592] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4066.839107] Lustre: Skipped 220 previous similar messages [ 4066.855646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.863740] Lustre: Skipped 220 previous similar messages [ 4066.867373] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4066.870984] Lustre: Skipped 220 previous similar messages [ 4066.874582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4066.882112] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4066.888868] Lustre: Skipped 220 previous similar messages [ 4067.420054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4067.972319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4068.477089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4069.017054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4069.556535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4070.086486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4070.589034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4071.100313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4071.707406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4072.202515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4072.661167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4073.207434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4073.647378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4074.130621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4074.699184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4075.199713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4075.660501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4076.169461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4076.704373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4077.227299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4077.782386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4078.340270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4078.887963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4079.500471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4080.097889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4080.661280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4081.180851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4081.763175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4082.341631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4082.952303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4083.476325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4084.111403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4084.675426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4085.252137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4085.824154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4086.371292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4086.911250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4087.527782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4088.031544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4088.486885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4089.031471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4089.584878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4090.106186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4090.676794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4091.204591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4091.769698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4092.368124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4092.852264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4093.398851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4093.912877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4094.412250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4094.918816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4095.406818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4095.917540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4096.491642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4097.015356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4097.550485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4098.125263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4098.658743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4099.192431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4099.793047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4100.384082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4100.938289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4101.526538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4102.025914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4102.499752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4102.961372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4103.470328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4103.998253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4104.526042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4105.077618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4105.590518] LustreError: 170854:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4105.599551] LustreError: 170854:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4105.600142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4106.063250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4106.564386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4107.150751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4107.701240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4108.263402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4108.844838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4109.392989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4109.928735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4110.407853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4110.898780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4111.428546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4111.908715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4112.441962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4112.962928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4113.488795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4113.986973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4114.482609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4115.012542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4115.514559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4116.072806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4116.569382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4117.095030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4117.681961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4118.222278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4118.731224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4119.292406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4119.842870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4120.393426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4120.946454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4121.544150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4122.168287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4122.749727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4123.353863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4123.973158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4124.574965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4125.083496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4125.608636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4126.131810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4126.678116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4127.290493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4127.797940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4128.358597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4128.904365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4129.439347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4129.923409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4130.402548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4130.873896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4131.399424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4131.925752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4132.415743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4132.882969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4133.413806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4133.890165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4134.355401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4134.837872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4135.355143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4135.888752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4136.350361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4136.864789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4137.383834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4137.946356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4138.428935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4138.892472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4139.406088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4139.834705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4140.341838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4140.844819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4141.356276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4141.883413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4142.450113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4142.922372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4143.459164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4144.018743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4144.496328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4144.980567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4145.496593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4145.940996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4146.393822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4146.863585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4147.338980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4147.797554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4148.306084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4148.789963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4149.272559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4149.707573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4150.171183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4150.673275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4151.175755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4151.757819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4152.409679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4152.971769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4153.509822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4154.116503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4154.681804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4155.263253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4155.809907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4156.359073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4156.920259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4157.475274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4158.060281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4158.690798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4159.249948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4159.935983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4160.439500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4160.954288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4161.632727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4162.142119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4162.710717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4163.296742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4163.857821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4164.430962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4165.108261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4165.689109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4166.268916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4166.824208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4167.396120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4167.906749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4168.397619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4168.873469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4169.395472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4169.926697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4170.384949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4170.901152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4171.414589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4171.935991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4172.466975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4172.987819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4173.543269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4174.026805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4174.494949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4174.970670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4175.481589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4175.953791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4176.453922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4176.912935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4177.355939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4177.831419] LustreError: 170854:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4177.835951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4177.836925] LustreError: 170854:0:(osp_precreate.c:1356:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4178.279580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4178.672577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4179.076938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4179.461963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4179.825950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4180.230434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4180.661457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4181.112154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4181.537965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4181.954588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4182.429037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4182.865933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4183.280903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4183.691426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4184.107301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4184.488852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4184.863550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4185.249753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4185.599690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4185.944376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4186.293747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4186.642343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4186.993954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4187.360527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4187.721597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4188.073086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4188.431688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4188.928610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4189.408113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4189.819499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4190.251567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4190.658413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4191.093163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4191.568713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4192.086710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4192.649814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4193.186619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4193.903530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4194.550564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4195.153407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4195.633596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4196.201896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4196.801682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4197.846580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4199.017614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4199.967757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4201.529645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4202.576642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4203.489320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4205.088898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4206.347419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4207.711251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4208.843152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4210.181372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4211.455886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4212.745312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4214.200865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4216.128573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4217.581865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4219.389441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4221.165317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4222.531058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4224.255038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.540495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4228.693167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4231.627688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4233.273974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4237.403758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4239.952796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4242.313075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4244.267638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4246.253903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4247.605179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4248.818474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4250.623189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4252.787891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4255.035433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4257.160192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4259.027059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4260.306037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4262.077962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4263.688534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4265.760098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4267.061996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4268.597312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4269.769994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4270.916668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4271.933715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4272.897728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4273.830854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4274.865952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4275.745979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4276.618979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4277.457408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4278.332799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4279.128245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4279.944128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4280.676994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4281.465551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.236088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.959580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4283.664466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4284.404429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4285.449717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4286.449409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4287.398237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4288.298618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.378843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4290.461814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4291.614223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4292.636462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4293.752881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4295.109029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4296.036130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4297.550393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4298.954415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4300.507789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4301.814872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4302.761969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4304.166893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4305.795328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4307.590887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4309.223699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4310.781715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4312.236638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4313.704124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4314.896545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4315.929524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4317.129680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4318.489570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4319.298883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4320.680146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4322.305730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4323.489777] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4323.496779] Lustre: Skipped 342 previous similar messages [ 4323.515367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.526927] Lustre: Skipped 342 previous similar messages [ 4323.540079] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4323.550841] Lustre: Skipped 342 previous similar messages [ 4323.562141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4323.584908] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4323.597884] Lustre: Skipped 342 previous similar messages [ 4324.459621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4325.491927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4326.705802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.974507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4329.004186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4330.009132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4330.980911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4331.973442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4332.960976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4333.985729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4335.159484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4336.190922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4337.334575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4338.194227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4339.235055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4340.240645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4341.258098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4342.195339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4343.237631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4344.123343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4345.163785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4346.018440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4347.123324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4347.999887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4349.060872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4349.963470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4350.866429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4351.804421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4352.944528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4354.007737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4354.964844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4355.955381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4356.912697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4357.861179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4358.925463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4359.829579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4360.810409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4361.764754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4362.730672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4363.679961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4364.740541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4365.777124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4366.886088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4367.749266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4368.779436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4369.634192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4370.605310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4371.607450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4372.703705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4373.622185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4374.593832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4375.473669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4376.564609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4377.542229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4378.542298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4379.405467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4380.551871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4381.596073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4382.569411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4383.544138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4384.604616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4385.513887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4386.650619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4387.566195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4388.842409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4390.017633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4390.973983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4392.065758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4393.267938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4394.163232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4395.241487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4396.054652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4397.096505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4398.178785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4399.291207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4400.193942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4400.976477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4401.805485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4402.949367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4403.888304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4404.721882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4405.529181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4406.415556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4407.372288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4408.146231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4409.008331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4409.817151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4410.704771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4411.795227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4412.785565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4413.747508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4414.653706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4415.856144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4416.921362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4417.784658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4418.844125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4419.775921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4420.607227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4421.512736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4422.416487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4423.358755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4424.239421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4425.399602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4426.230757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4427.221946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4428.160373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4429.270276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4430.059839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4430.875968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4431.667269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4432.528634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4433.599854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4434.442974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4435.478606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4436.374636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4437.679397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4438.570928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4439.547937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4440.692510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4441.550541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4442.664903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4443.733296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4444.895495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4446.178747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4447.320603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4448.476430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4449.196385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4450.310886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4451.576182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4452.609128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4453.472420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4454.530149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4455.624357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4456.722896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4457.846516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4459.054067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4460.302483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4461.496775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4462.508481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4463.506966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4464.714329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4465.743934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4466.963690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4468.286505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4469.264878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4470.622525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4471.618870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4472.793259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4473.898225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4474.929545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4475.904445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4476.982462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4477.720061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4478.625448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4479.459114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4480.497740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4481.732980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4482.840091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4483.772887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4485.021418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4485.766741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4486.716055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4487.743791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4488.770907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4489.770811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4491.196500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4492.125419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4493.451656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.678385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4495.885327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4497.317376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4498.574305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4499.839986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4500.757505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4501.881791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4502.934235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4504.098649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4504.948066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4505.866279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4506.706891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4507.727958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4508.628042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4509.436067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4510.255781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.217797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4512.034832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4512.860320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4513.855591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4515.145876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4516.209900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4517.409173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4518.488550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.587809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4520.652772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4521.720877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4522.689992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4523.754059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4524.845123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.927582] LustreError: 172656:0:(ofd_dev.c:1782:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 5118.955228] LustreError: 170854:0:(osp_precreate.c:652:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6006.241377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6006.253449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.269922] Lustre: Skipped 198 previous similar messages [ 6006.294418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.316403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.327791] Lustre: Skipped 1 previous similar message [ 6011.689146] Lustre: server umount lustre-OST0000 complete [ 6015.968264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6015.978547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6019.556068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6019.560528] Lustre: Skipped 1 previous similar message [ 6021.985762] Lustre: server umount lustre-MDT0000 complete [ 6024.676729] LustreError: 170835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6024.706854] LustreError: 170835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6026.186028] LustreError: 170816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490920 with bad export cookie 17822607159674299081 [ 6026.205489] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6026.207552] LustreError: 170816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6026.690442] Lustre: server umount lustre-MDT0001 complete [ 6040.500172] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6044.141539] Key type lgssc unregistered [ 6044.479558] LNet: 226143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6044.486324] LNetError: 226143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6044.504047] LNet: Removed LNI 192.168.204.154@tcp [ 6045.449256] Key type .llcrypt unregistered [ 6045.453357] Key type ._llcrypt unregistered [ 6077.619920] Key type ._llcrypt registered [ 6077.622483] Key type .llcrypt registered [ 6077.736922] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6079.228663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6079.349355] alg: No test for adler32 (adler32-zlib) [ 6080.649340] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 6081.196435] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6083.097404] Key type lgssc registered [ 6084.750868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6093.877954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6093.894498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.643165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6101.084632] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6106.100393] LustreError: 227562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6106.126167] LustreError: 227562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6111.203791] LustreError: 227563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6111.418261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.766343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6118.342927] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6132.506096] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6146.264940] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6155.655313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6156.152247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.927168] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6166.430817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6174.228460] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6181.619042] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6184.466653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6184.475055] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6184.479774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.489844] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6184.491540] LustreError: 227582:0:(osp_precreate.c:1264:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6187.394787] Lustre: lustre-OST0000: Not available for connect from 192.168.204.54@tcp (stopping) [ 6188.006410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6188.017620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6188.068228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6192.098767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.134564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6192.149528] Lustre: Skipped 1 previous similar message [ 6197.242321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6197.272798] Lustre: Skipped 2 previous similar messages [ 6201.311550] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6201.600287] Lustre: server umount lustre-OST0000 complete [ 6204.597310] LustreError: 227582:0:(osp_precreate.c:1264:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6243.854533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6244.938803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6246.960260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.974466] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6252.130608] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.146548] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6254.968818] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6266.055381] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6268.721990] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6277.293532] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d92062e9800.ost_server_uuid 50 [ 6280.391803] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d92062e9800.ost_server_uuid in IDLE state after 0 sec [ 6292.366043] Lustre: server umount lustre-OST0000 complete [ 6301.170111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6301.191280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6301.211208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6301.222217] Lustre: Skipped 1 previous similar message [ 6309.855182] Lustre: 226874:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782491188/real 1782491188] req@00000000af2703c6 x1869077256682240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782491204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6309.865263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.917649] Lustre: 226874:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6309.927220] Lustre: Skipped 1 previous similar message [ 6309.931249] LustreError: 227558:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.975794] LustreError: 227558:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6310.035908] Lustre: server umount lustre-MDT0000 complete [ 6314.981211] LustreError: 228956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6314.993133] LustreError: 228956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6315.040254] Lustre: 226872:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782491193/real 1782491193] req@000000002bba299d x1869077256683392/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782491209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6315.227690] LustreError: 227544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491209 with bad export cookie 7302498237804283949 [ 6315.237136] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6315.248396] LustreError: 227544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6315.667098] Lustre: server umount lustre-MDT0001 complete [ 6328.819320] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 6332.520298] Key type lgssc unregistered [ 6332.891777] LNet: 232779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6332.908254] LNetError: 232779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6332.936994] LNet: Removed LNI 192.168.204.154@tcp [ 6333.924485] Key type .llcrypt unregistered [ 6333.932734] Key type ._llcrypt unregistered [ 6370.562776] Key type ._llcrypt registered [ 6370.568427] Key type .llcrypt registered [ 6370.694989] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6371.873860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6371.956512] alg: No test for adler32 (adler32-zlib) [ 6373.178711] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 6373.566812] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 6375.279158] Key type lgssc registered [ 6376.951617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6388.401749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6388.417795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.056282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.205433] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6399.457574] LustreError: 234511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.482984] LustreError: 234511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6404.584380] LustreError: 234512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6406.470454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.804308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6411.419052] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6414.331434] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6421.101576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6421.407152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6427.684430] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6435.796581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 6436.494181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6436.711438] Lustre: lustre-OST0001: new disk, initializing [ 6436.717370] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6436.724473] Lustre: 236817:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6436.825061] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6444.089682] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6444.111048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6444.115955] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6444.223582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6455.230062] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6462.471288] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6475.234414] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.245307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6475.245641] Lustre: Skipped 3 previous similar messages [ 6475.262443] Lustre: Skipped 3 previous similar messages [ 6480.362313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6480.369106] Lustre: Skipped 3 previous similar messages [ 6481.436906] Lustre: server umount lustre-MDT0000 complete [ 6484.921230] LustreError: 234492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491379 with bad export cookie 12184116894228991271 [ 6484.933060] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6484.937789] LustreError: 234492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6485.329309] Lustre: server umount lustre-MDT0001 complete [ 6499.348060] Lustre: server umount lustre-OST0000 complete [ 6501.855169] Lustre: 233775:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782491380/real 1782491380] req@ffff98dc195bea00 x1869077563926528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782491396 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6501.881459] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6503.239440] Lustre: server umount lustre-OST0001 complete [ 6525.164768] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6534.788060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.356841] LustreError: 239925:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.481227] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6539.589900] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6540.770963] LustreError: 239926:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6544.879155] LustreError: 239925:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6548.134120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.630589] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6553.235868] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6556.021140] Lustre: 241066:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6563.005765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6570.180368] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6573.568409] LustreError: 241421:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6573.607346] LustreError: 241421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6579.704058] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 6579.725650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.994671] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6580.005905] Lustre: Skipped 1 previous similar message [ 6581.002229] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6587.186466] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6594.801568] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6598.222931] Lustre: 242937:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6609.891841] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6609.909181] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.928363] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.944942] Lustre: mylustre-MDT0000-lwp-OST0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.951576] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.967868] Lustre: Skipped 2 previous similar messages [ 6611.982344] Lustre: Skipped 3 previous similar messages [ 6614.663990] Lustre: server umount mylustre-MDT0000 complete [ 6617.071414] LustreError: 240658:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.081534] LustreError: 240658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6618.938493] LustreError: 239901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491513 with bad export cookie 12184116894228994078 [ 6618.940182] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6618.958669] LustreError: 239901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6619.500196] Lustre: server umount mylustre-MDT0001 complete [ 6634.075222] Lustre: server umount mylustre-OST0000 complete [ 6637.535316] Lustre: 233776:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782491516/real 1782491516] req@ffff98dd26e90e00 x1869077563988352/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782491532 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6637.570708] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6638.338464] Lustre: server umount mylustre-OST0001 complete [ 6661.281756] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6672.184159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6672.600299] LustreError: 245243:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.687369] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6677.406725] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6686.372866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.868195] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6694.952145] Lustre: 246388:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6702.166105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.898883] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6717.863321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6717.940426] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49792) [ 6723.577632] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6724.354126] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6731.642821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6735.427863] Lustre: 248258:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6743.009846] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6743.015799] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6743.033203] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 6747.939394] Lustre: server umount tfs-MDT0000 complete [ 6749.160570] LustreError: 245240:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6749.183027] LustreError: 245240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6751.165877] LustreError: 245218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491645 with bad export cookie 12184116894228995968 [ 6751.168472] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6751.172802] LustreError: 245218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6751.424213] Lustre: server umount tfs-MDT0001 complete [ 6765.277739] Lustre: server umount tfs-OST0000 complete [ 6778.951349] Lustre: server umount tfs-OST0001 complete [ 6798.129950] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6806.197266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6806.725616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6806.730341] Lustre: Skipped 3 previous similar messages [ 6810.954531] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6819.325407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6823.530734] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6826.207812] Lustre: 251649:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6833.169128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6839.806708] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6848.096602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6852.409282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49824) [ 6852.416488] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6853.701301] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6860.290933] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6863.717718] Lustre: 253518:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6870.235948] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 12:36:02 (1782491762) [ 6886.371257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6886.379378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.391665] Lustre: Skipped 3 previous similar messages [ 6886.396288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6886.399089] Lustre: Skipped 4 previous similar messages [ 6893.538517] LustreError: 251240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6893.557710] LustreError: 251240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6893.956505] Lustre: server umount lustre-MDT0000 complete [ 6896.606150] LustreError: 250482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491791 with bad export cookie 12184116894228997634 [ 6896.614678] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6896.625400] LustreError: 250482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6902.935714] Lustre: server umount lustre-MDT0001 complete [ 6912.293745] Lustre: server umount lustre-OST0000 complete [ 6930.399779] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6930.548259] Lustre: server umount lustre-OST0001 complete [ 6936.216448] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 6944.277844] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 6975.509713] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 6975.509713] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6975.509713] [ 6975.543394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6975.825914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6975.881207] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6975.926784] Lustre: lustre-MDT0000: new disk, initializing [ 6976.007550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6976.011389] Lustre: Skipped 3 previous similar messages [ 6976.028430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6980.488114] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 6990.092912] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 6990.092912] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6990.092912] [ 6990.112042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6990.235294] Lustre: 256702:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6990.282696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6990.287972] Lustre: Skipped 1 previous similar message [ 6990.348770] Lustre: lustre-MDT0001: new disk, initializing [ 6990.477886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6990.487874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6994.325592] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7002.884305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7003.162285] Lustre: lustre-OST0000: new disk, initializing [ 7003.165781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7003.173804] Lustre: 258283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7005.067441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7005.075433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7005.150269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7009.260990] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7018.375250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.572714] Lustre: lustre-OST0001: new disk, initializing [ 7018.577858] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7018.585853] Lustre: 259158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7019.736281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7019.747553] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7019.894505] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7024.952614] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7035.345982] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 12:38:48 (1782491928) [ 7048.497453] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 12:39:01 (1782491941) [ 7066.084259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7066.100217] Lustre: Skipped 5 previous similar messages [ 7066.105263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7066.111884] Lustre: Skipped 9 previous similar messages [ 7068.825735] Lustre: server umount lustre-OST0000 complete [ 7078.987314] Lustre: server umount lustre-MDT0000 complete [ 7082.653160] LustreError: 256695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491977 with bad export cookie 12184116894228999188 [ 7082.662746] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7082.666224] LustreError: 256695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7089.009647] Lustre: server umount lustre-MDT0001 complete [ 7098.687089] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7114.812453] Lustre: server umount lustre-OST0001 complete [ 7120.650169] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7132.912400] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7162.923268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7163.094815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7163.112847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7163.162457] Lustre: lustre-MDT0000: new disk, initializing [ 7163.243976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7166.790289] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7176.398843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.562542] Lustre: 263693:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7176.822158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7180.310473] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7188.119282] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7192.592622] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7198.124226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7198.324236] Lustre: lustre-OST0000: new disk, initializing [ 7198.326845] Lustre: Skipped 1 previous similar message [ 7198.330578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7198.334345] Lustre: Skipped 2 previous similar messages [ 7198.339220] Lustre: 265660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7200.450105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7200.462354] Lustre: Skipped 1 previous similar message [ 7200.467386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7200.571404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7203.132583] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7210.625575] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7216.097845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7216.100217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.107274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7216.112623] Lustre: Skipped 6 previous similar messages [ 7216.119749] Lustre: Skipped 6 previous similar messages [ 7221.415614] Lustre: server umount lustre-OST0000 complete [ 7230.346603] Lustre: server umount lustre-MDT0000 complete [ 7231.466489] LustreError: 263705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7231.486228] LustreError: 263705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 7233.436762] LustreError: 263686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492128 with bad export cookie 12184116894229000658 [ 7233.441774] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7233.451654] LustreError: 263686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7233.740111] Lustre: server umount lustre-MDT0001 complete [ 7241.782399] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7243.821561] Key type lgssc unregistered [ 7244.114636] LNet: 267417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7244.123184] LNetError: 267417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7244.133857] LNet: Removed LNI 192.168.204.154@tcp [ 7244.763175] Key type .llcrypt unregistered [ 7244.772343] Key type ._llcrypt unregistered [ 7263.353282] Key type ._llcrypt registered [ 7263.354971] Key type .llcrypt registered [ 7263.437252] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7264.022851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7264.038237] alg: No test for adler32 (adler32-zlib) [ 7265.020301] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 7265.233838] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7266.871161] Key type lgssc registered [ 7267.733984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7273.646477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7273.661740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7275.065153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7278.523362] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7284.648589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7284.910759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7288.073155] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7293.679109] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7297.724942] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7302.413212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7302.612756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7307.000861] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7312.323526] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7313.162819] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 7314.915069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7314.929136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7314.957740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7316.754827] Lustre: server umount lustre-OST0000 complete [ 7320.033515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7320.039669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7320.050541] Lustre: Skipped 1 previous similar message [ 7320.062818] Lustre: Skipped 1 previous similar message [ 7325.154261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7325.161437] Lustre: Skipped 1 previous similar message [ 7325.859816] Lustre: server umount lustre-MDT0000 complete [ 7328.661305] LustreError: 268839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492223 with bad export cookie 13047608416887531306 [ 7328.663348] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7328.670587] LustreError: 268839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7328.936217] Lustre: server umount lustre-MDT0001 complete [ 7336.543977] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7338.513413] Key type lgssc unregistered [ 7338.721485] LNet: 272441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7338.730715] LNetError: 272441:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7338.747116] LNet: Removed LNI 192.168.204.154@tcp [ 7339.410089] Key type .llcrypt unregistered [ 7339.413439] Key type ._llcrypt unregistered [ 7351.706854] Key type ._llcrypt registered [ 7351.709478] Key type .llcrypt registered [ 7351.792827] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7352.705970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7352.727085] alg: No test for adler32 (adler32-zlib) [ 7353.780764] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 7353.947092] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7355.591226] Key type lgssc registered [ 7356.470768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7362.677491] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7362.692759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.271353] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7383.391186] LustreError: 273615:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.154@tcp: failed processing log, type 4: rc = -110 [ 7410.017596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7413.845519] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7418.875809] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7420.383901] Lustre: server umount lustre-OST0000 complete [ 7431.597585] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7433.466979] Key type lgssc unregistered [ 7433.688801] LNet: 275243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7433.693806] LNetError: 275243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7433.703487] LNet: Removed LNI 192.168.204.154@tcp [ 7434.209024] Key type .llcrypt unregistered [ 7434.212532] Key type ._llcrypt unregistered [ 7459.062424] Key type ._llcrypt registered [ 7459.065375] Key type .llcrypt registered [ 7459.137720] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7459.824709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7459.837637] alg: No test for adler32 (adler32-zlib) [ 7460.805613] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 7460.951094] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 7462.567197] Key type lgssc registered [ 7463.254785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7471.697639] loop0: detected capacity change from 0 to 204800000 [ 7471.793566] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7471.801744] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7471.996800] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7472.597313] loop0: detected capacity change from 0 to 204800000 [ 7472.702502] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7472.712180] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7472.915550] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7473.573283] loop0: detected capacity change from 0 to 204800000 [ 7473.699452] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7473.705721] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7473.777844] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7474.466824] loop0: detected capacity change from 0 to 204800000 [ 7474.559915] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7475.318344] loop0: detected capacity change from 0 to 204800000 [ 7475.345367] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7478.212413] loop0: detected capacity change from 0 to 204800000 [ 7478.243505] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7480.996382] loop0: detected capacity change from 0 to 204800000 [ 7481.020866] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7483.758734] loop0: detected capacity change from 0 to 204800000 [ 7483.780834] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 7486.836391] loop0: detected capacity change from 0 to 204800000 [ 7486.926100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7486.950159] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7487.024561] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7487.037859] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 7487.047406] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7487.064255] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7487.080252] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 7487.090421] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 7487.099936] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7487.110648] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 7487.120627] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 7487.129107] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 7487.140532] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 7487.151810] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 7487.163811] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 7487.174109] Lustre: 278635:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 7491.323152] Lustre: server umount MGS complete [ 7491.947665] loop0: detected capacity change from 0 to 204800000 [ 7491.983324] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.277539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.303634] Lustre: lustre-MDT0000: Aborting client recovery [ 7492.915349] loop1: detected capacity change from 0 to 204800000 [ 7492.996833] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7493.073329] Lustre: 279067:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7493.085219] Lustre: 279067:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 7493.096730] Lustre: 279067:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 7493.107628] Lustre: 279067:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7493.121582] Lustre: 279067:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7493.147986] Lustre: 279067:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7493.160988] Lustre: 279067:0:(osd_scrub.c:1830:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 7493.181299] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7493.297270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7493.339464] Lustre: lustre-MDT0001: Aborting client recovery [ 7493.908815] loop2: detected capacity change from 0 to 204800000 [ 7494.007681] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7494.053111] Lustre: 279146:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7494.061546] Lustre: 279146:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 7494.072130] Lustre: 279146:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 7494.082780] Lustre: 279146:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7494.091291] Lustre: 279146:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7494.098746] Lustre: 279146:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 7494.106149] Lustre: 279146:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7494.195280] Lustre: Found index 0 for lustre-OST0000, updating log [ 7494.232723] Lustre: 279164:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7494.288763] Lustre: lustre-OST0000: Aborting recovery [ 7494.939709] loop3: detected capacity change from 0 to 204800000 [ 7495.050862] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.102662] Lustre: 279228:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 7495.109201] Lustre: 279228:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 7495.126850] Lustre: 279228:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 7495.143206] Lustre: 279228:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 7495.152141] Lustre: 279228:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 7495.159566] Lustre: 279228:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 7495.169995] Lustre: 279228:0:(osd_scrub.c:1830:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 7495.208508] Lustre: 279234:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7495.250124] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7495.255183] Lustre: Skipped 1 previous similar message [ 7495.275278] Lustre: lustre-OST0001: Aborting recovery [ 7500.277934] LustreError: 279010:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -108 [ 7502.310966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7502.317284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7502.339364] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 7502.347445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 7503.333890] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 7503.337930] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 7508.187048] LustreError: 278977:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1459us: rc = -2 [ 7508.692328] LustreError: 278977:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 383us: rc = -2 [ 7508.705632] LustreError: 278977:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 36 previous similar messages [ 7523.812499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.812845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7523.827647] Lustre: Skipped 2 previous similar messages [ 7523.837506] Lustre: Skipped 2 previous similar messages [ 7526.312507] Lustre: server umount lustre-MDT0000 complete [ 7526.813725] LustreError: 278962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492421 with bad export cookie 1182991048788833633 [ 7526.826538] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7528.930787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7528.931598] LustreError: 278976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7528.936147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7528.941429] Lustre: Skipped 2 previous similar messages [ 7528.958991] LustreError: 278976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7531.939085] LustreError: 278975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7531.939378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7531.954572] LustreError: 278975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7531.960168] Lustre: Skipped 2 previous similar messages [ 7537.056514] LustreError: 278976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.056636] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7537.066622] LustreError: 278976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7537.073650] Lustre: Skipped 1 previous similar message [ 7541.215127] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7541.350373] Lustre: server umount lustre-MDT0001 complete [ 7547.996660] Lustre: server umount lustre-OST0000 complete [ 7554.622086] Lustre: server umount lustre-OST0001 complete [ 7563.898207] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 12:47:36 (1782492456) [ 7601.733744] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 7615.947760] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 7659.382538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7659.666046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7659.715857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7659.804416] Lustre: lustre-MDT0000: new disk, initializing [ 7659.891494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.910182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7664.886548] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7675.388083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.472807] Lustre: 283301:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7675.518567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7675.524628] Lustre: Skipped 1 previous similar message [ 7675.604531] Lustre: lustre-MDT0001: new disk, initializing [ 7675.732498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7675.777918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7675.794767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7679.250506] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7687.845443] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7693.543116] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7699.749659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7700.121372] Lustre: lustre-OST0000: new disk, initializing [ 7700.126275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7700.135838] Lustre: 285269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7700.223936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7707.456896] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7708.768487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7708.792052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7708.931065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7716.566876] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7721.421502] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 7735.461649] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 7741.923282] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 7785.956468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7785.959774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7785.986403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7785.988745] LustreError: Skipped 1 previous similar message [ 7792.044627] Lustre: server umount lustre-OST0000 complete [ 7796.193803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7796.207411] Lustre: Skipped 1 previous similar message [ 7796.216229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7796.222046] Lustre: Skipped 4 previous similar messages [ 7802.161786] Lustre: server umount lustre-MDT0000 complete [ 7805.842642] LustreError: 283295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492700 with bad export cookie 1182991048788836342 [ 7805.848050] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7805.855251] LustreError: 283295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7806.201551] Lustre: server umount lustre-MDT0001 complete [ 7814.693773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.150305] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7824.882473] Lustre: server umount MGS complete [ 7836.405550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.130892] LustreError: 289184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.161764] LustreError: 289197:0:(qmt_pool.c:1630:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7837.209412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7841.835781] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7842.273782] LustreError: 289185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7852.267331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7852.514056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7852.530344] Lustre: Skipped 2 previous similar messages [ 7858.313940] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7867.258231] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7872.884599] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7879.445071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7879.732774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7879.738305] Lustre: Skipped 1 previous similar message [ 7885.680746] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 7893.086345] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7922.657721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7922.670137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.693389] Lustre: Skipped 1 previous similar message [ 7922.709144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7923.683577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7926.773726] Lustre: server umount lustre-OST0000 complete [ 7933.927456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.949545] Lustre: Skipped 2 previous similar messages [ 7936.737989] Lustre: server umount lustre-MDT0000 complete [ 7939.046494] LustreError: 289184:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7939.064249] LustreError: 289184:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7940.823540] LustreError: 289166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492835 with bad export cookie 1182991048788838176 [ 7940.836092] LustreError: 289166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7940.837153] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7941.148311] Lustre: server umount lustre-MDT0001 complete [ 7953.118471] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 7956.135514] Key type lgssc unregistered [ 7956.356167] LNet: 293127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7956.366133] LNetError: 293127:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7956.382787] LNet: Removed LNI 192.168.204.154@tcp [ 7957.345701] Key type .llcrypt unregistered [ 7957.348675] Key type ._llcrypt unregistered [ 7997.977814] Key type ._llcrypt registered [ 7997.979725] Key type .llcrypt registered [ 7998.084529] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 8016.270718] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8017.149635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8017.191964] alg: No test for adler32 (adler32-zlib) [ 8018.261644] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 8018.594098] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8020.319173] Key type lgssc registered [ 8021.720213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8060.225145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8060.288490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.538115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8061.571366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8061.675076] Lustre: lustre-MDT0000: new disk, initializing [ 8061.791451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8061.816178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8066.667706] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8075.915909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8075.985245] Lustre: 297095:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8076.035123] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8076.038374] Lustre: Skipped 1 previous similar message [ 8076.101218] Lustre: lustre-MDT0001: new disk, initializing [ 8076.177412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8076.214903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8076.221844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8080.594540] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8091.020274] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8097.856914] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8105.797760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8106.094433] Lustre: lustre-OST0000: new disk, initializing [ 8106.099451] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8106.117069] Lustre: 299061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8106.223108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.352308] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8114.734522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8114.748235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8114.794955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8121.351642] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8127.522468] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8139.755760] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8146.042798] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8151.833342] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8191.456175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8191.458582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8191.477428] Lustre: Skipped 1 previous similar message [ 8196.585696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8196.590544] Lustre: Skipped 2 previous similar messages [ 8197.535497] Lustre: server umount lustre-OST0000 complete [ 8201.697455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8201.702146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8201.727289] Lustre: Skipped 1 previous similar message [ 8206.816523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8206.833591] Lustre: Skipped 2 previous similar messages [ 8207.926834] Lustre: server umount lustre-MDT0000 complete [ 8212.849385] LustreError: 297087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782493107 with bad export cookie 15789456424251918369 [ 8212.860165] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8212.868256] LustreError: 297087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8213.660949] Lustre: server umount lustre-MDT0001 complete [ 8223.955803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.127157] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8233.549125] Lustre: server umount MGS complete [ 8245.663840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.353495] LustreError: 302934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.366753] LustreError: 302934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8246.387708] LustreError: 302947:0:(qmt_pool.c:1630:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8246.505381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.536965] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8251.887776] LustreError: 302935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8255.979902] LustreError: 302934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8261.090340] LustreError: 302935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8261.256572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8261.860548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8266.760921] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8274.801572] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8281.507410] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8287.953929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8288.292823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.560193] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8302.316175] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8333.283772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8333.307592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8333.352762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8333.816353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8338.188600] Lustre: server umount lustre-OST0000 complete [ 8343.520478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8343.527958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.537993] Lustre: Skipped 1 previous similar message [ 8343.546755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8343.553724] Lustre: Skipped 1 previous similar message [ 8349.061822] Lustre: server umount lustre-MDT0000 complete [ 8349.153465] LustreError: 302934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8349.170466] LustreError: 302934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8353.152212] LustreError: 305772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782493247 with bad export cookie 15789456424251920294 [ 8353.166046] LustreError: 305772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8353.168330] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8353.521328] Lustre: server umount lustre-MDT0001 complete [ 8366.560468] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 8371.288734] Key type lgssc unregistered [ 8371.750736] LNet: 306893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8371.765566] LNetError: 306893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8371.809233] LNet: Removed LNI 192.168.204.154@tcp [ 8373.128119] Key type .llcrypt unregistered [ 8373.132367] Key type ._llcrypt unregistered [ 8398.668821] Key type ._llcrypt registered [ 8398.673041] Key type .llcrypt registered [ 8398.856624] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 8414.891488] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8416.058164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8416.177910] alg: No test for adler32 (adler32-zlib) [ 8417.263872] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 8417.465233] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 8419.176622] Key type lgssc registered [ 8420.478330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8446.500104] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 8462.214034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8462.262405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8463.651091] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8463.709832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8463.788183] Lustre: lustre-MDT0000: new disk, initializing [ 8463.893190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8463.928549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8468.801537] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8480.707652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8481.076947] Lustre: lustre-OST0000: new disk, initializing [ 8481.080497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8481.085887] Lustre: Skipped 1 previous similar message [ 8481.091424] Lustre: 310946:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8481.182579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8488.581134] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 8490.162083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8490.173958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8490.212156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9539.556578] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9539.597566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9540.618574] Lustre: server umount lustre-MDT0000 complete [ 9545.239376] LustreError: 310131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494439 with bad export cookie 10842083689143521117 [ 9545.263152] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9545.661972] Lustre: server umount lustre-OST0000 complete [ 9583.878293] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [ 9598.563678] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9647.172816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9647.545103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9647.593743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9647.695465] Lustre: lustre-MDT0000: new disk, initializing [ 9647.806134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9647.924050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9653.453578] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9666.004638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9666.123995] Lustre: 315331:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9666.216498] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9666.222137] Lustre: Skipped 1 previous similar message [ 9666.342581] Lustre: lustre-MDT0001: new disk, initializing [ 9666.495605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9666.542593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9666.556990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9672.243826] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9684.181433] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9693.013390] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9702.004075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9702.438495] Lustre: lustre-OST0000: new disk, initializing [ 9702.450564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9702.459525] Lustre: 317300:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9702.601034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9710.670962] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9712.731092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9712.745574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9712.914959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9722.460419] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9728.086054] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9728.317453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9733.950305] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9734.237991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9738.728865] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9738.729284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.738515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9738.741452] LustreError: Skipped 1 previous similar message [ 9738.770094] Lustre: Skipped 1 previous similar message [ 9742.838617] Lustre: server umount lustre-OST0000 complete [ 9748.450637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9748.463835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.480298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9748.484831] Lustre: Skipped 1 previous similar message [ 9753.483804] Lustre: server umount lustre-MDT0000 complete [ 9758.090804] LustreError: 315323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494652 with bad export cookie 10842083689146461929 [ 9758.108094] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9758.113764] LustreError: 315323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9758.608137] Lustre: server umount lustre-MDT0001 complete [ 9762.791789] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 9764.735637] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 13:24:17 (1782494657) [ 9772.344414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9772.763208] LustreError: 319613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9772.786280] LustreError: 319613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9772.845791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9778.148469] LustreError: 319614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9778.455698] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9783.266288] LustreError: 319613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9788.189786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9788.387093] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9788.398504] Lustre: Skipped 1 previous similar message [ 9788.515904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9793.741852] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9801.734405] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9808.213871] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9815.877488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9816.332440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9824.326637] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9833.888272] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9842.152434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9842.337975] Lustre: lustre-OST0001: new disk, initializing [ 9842.341343] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9842.351363] Lustre: 322449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9842.490221] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9849.858564] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [ 9851.047367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9851.061761] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9851.149174] LustreError: 322847:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9852.194846] LustreError: 322847:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9852.202594] LustreError: 322847:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 9853.236072] LustreError: 322847:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9853.257061] LustreError: 322847:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 9855.334341] LustreError: 322847:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9855.362080] LustreError: 322847:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [ 9859.441476] LustreError: 322864:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 9859.460505] LustreError: 322864:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [ 9861.755410] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9862.093313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9867.700519] LustreError: 322847:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 9867.718345] LustreError: 322847:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [ 9868.709703] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d92062e8800.ost_server_uuid 50 [ 9870.776662] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d92062e8800.ost_server_uuid in FULL state after 0 sec [ 9879.001714] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d92062e8800.ost_server_uuid 50 [ 9881.180799] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d92062e8800.ost_server_uuid in IDLE state after 0 sec [ 9883.344201] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9897.446501] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.453434] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9897.464898] Lustre: Skipped 2 previous similar messages [ 9903.420724] Lustre: server umount lustre-OST0001 complete [ 9907.405131] LustreError: 321434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9910.753860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9910.758641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9910.778128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9910.787968] Lustre: Skipped 5 previous similar messages [ 9912.815232] LustreError: 321435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.842818] LustreError: 321435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9915.850523] Lustre: server umount lustre-OST0000 complete [ 9921.505741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9921.516078] LustreError: Skipped 1 previous similar message [ 9921.521560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.550240] Lustre: Skipped 1 previous similar message [ 9926.906613] Lustre: server umount lustre-MDT0000 complete [ 9928.168481] LustreError: 319614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.203268] LustreError: 319614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9931.681885] LustreError: 319596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494826 with bad export cookie 10842083689146462818 [ 9931.686611] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9931.693489] LustreError: 319596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9932.218909] Lustre: server umount lustre-MDT0001 complete [ 9947.552953] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [ 9952.173301] Key type lgssc unregistered [ 9952.768170] LNet: 325147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9952.791157] LNetError: 325147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9952.830527] LNet: Removed LNI 192.168.204.154@tcp [ 9953.847326] Key type .llcrypt unregistered [ 9953.849465] Key type ._llcrypt unregistered [ 9982.784772] Key type ._llcrypt registered [ 9982.787337] Key type .llcrypt registered [ 9982.915890] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [ 9983.934351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9984.031884] alg: No test for adler32 (adler32-zlib) [ 9985.287648] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 9985.557517] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [ 9987.239157] Key type lgssc registered [ 9988.466934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9996.260455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9996.308243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9997.850086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10003.476312] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10008.043809] LustreError: 326570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10008.081330] LustreError: 326570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10013.174199] LustreError: 326569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10014.811456] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10018.277588] LustreError: 326570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10022.195160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.765694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10028.633670] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10039.633451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10040.239444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10044.324533] LustreError: 328250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10044.358337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10047.748266] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10048.419904] LustreError: 328250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10048.453590] LustreError: 328250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10056.340931] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10058.724605] LustreError: 328250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10058.742888] LustreError: 328250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10063.976069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10064.173647] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10069.493986] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10070.770387] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10079.667979] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10086.520968] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10086.827375] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10202.595396] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10202.601699] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10202.611250] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10202.621973] Lustre: Skipped 1 previous similar message [10202.629404] Lustre: Skipped 1 previous similar message [10204.627552] Lustre: server umount lustre-OST0001 complete [10207.721932] LustreError: 329266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.739964] LustreError: 329266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10212.335225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10212.345829] LustreError: Skipped 1 previous similar message [10212.351160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10212.364626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10217.966848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10217.971798] Lustre: Skipped 2 previous similar messages [10223.077848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10223.094568] Lustre: Skipped 1 previous similar message [10225.121423] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10225.529674] Lustre: server umount lustre-OST0000 complete [10232.802861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10232.813556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10232.839204] Lustre: Skipped 1 previous similar message [10232.851617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10236.431378] Lustre: server umount lustre-MDT0000 complete [10241.569422] LustreError: 330256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495136 with bad export cookie 5303829478260443743 [10241.571925] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10241.589261] LustreError: 330256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10242.216848] Lustre: server umount lustre-MDT0001 complete [10254.632922] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [10257.867751] Key type lgssc unregistered [10258.133044] LNet: 331520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10258.137879] LNetError: 331520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10258.159836] LNet: Removed LNI 192.168.204.154@tcp [10259.040889] Key type .llcrypt unregistered [10259.050223] Key type ._llcrypt unregistered [10303.347732] Key type ._llcrypt registered [10303.350289] Key type .llcrypt registered [10303.479441] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10304.492050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10304.517956] alg: No test for adler32 (adler32-zlib) [10305.789697] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [10306.108262] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [10307.783801] Key type lgssc registered [10309.019316] Lustre: Echo OBD driver; http://www.lustre.org/ [10317.387210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10317.404828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10318.869322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10322.943320] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10329.062041] LustreError: 333813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10329.079392] LustreError: 333813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10331.151887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10331.471971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10335.852505] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10338.735986] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10345.299940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10345.635640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10351.806878] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10356.897614] LustreError: 335302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10360.951885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10361.203249] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10364.291624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10364.308429] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10368.557902] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10377.262598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10386.481696] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10400.229186] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10400.234056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10400.250408] Lustre: Skipped 3 previous similar messages [10400.264636] Lustre: Skipped 3 previous similar messages [10405.001514] Lustre: server umount lustre-MDT0000 complete [10405.350252] LustreError: 333814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.369031] LustreError: 333814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10408.695162] LustreError: 333793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495303 with bad export cookie 2901074163579000327 [10408.705517] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10408.711970] LustreError: 333793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10409.101971] Lustre: server umount lustre-MDT0001 complete [10423.351717] Lustre: server umount lustre-OST0000 complete [10426.851424] Lustre: 333076:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782495305/real 1782495305] req@ffff98dc0831d180 x1869081686899072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782495321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10426.887489] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.590545] Lustre: server umount lustre-OST0001 complete [10446.606830] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10457.883297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10458.257876] LustreError: 339076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10458.331707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10462.893900] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10463.726680] LustreError: 339077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10471.425824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.819490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10476.569406] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10479.655881] Lustre: 340216:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10486.108058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10492.229777] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10496.760630] LustreError: 340572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10496.796740] LustreError: 340572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10501.823505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10502.126848] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10502.139032] Lustre: Skipped 1 previous similar message [10503.183808] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [10503.200288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10509.806065] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10518.080247] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10522.580769] Lustre: 342089:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10528.231499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10528.246342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10528.265623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10533.401748] Lustre: server umount lustre-MDT0000 complete [10533.857614] LustreError: 339808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10533.883712] LustreError: 339808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10537.309271] LustreError: 340219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495431 with bad export cookie 2901074163579002028 [10537.326476] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10537.333186] LustreError: 340219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10537.792787] Lustre: server umount lustre-MDT0001 complete [10552.297127] Lustre: server umount lustre-OST0000 complete [10555.360318] Lustre: 333075:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782495433/real 1782495433] req@ffff98dc10ec0a80 x1869081686947968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782495449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10555.398330] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10555.414077] Lustre: Skipped 3 previous similar messages [10556.864385] Lustre: server umount lustre-OST0001 complete [10560.901990] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [10562.710904] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [10564.605196] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 13:37:37 (1782495457) [10589.832678] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10599.884839] loop0: detected capacity change from 0 to 1099511627776 [10600.742358] SGI XFS with ACLs, security attributes, quota, no debug enabled [10600.753076] *** VALIDATE xfs *** [10600.762262] XFS (loop0): Mounting V5 Filesystem [10600.783323] XFS (loop0): Ending clean mount [10602.188619] loop1: detected capacity change from 0 to 18691697672192 [10720.254892] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [10731.701983] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [10769.370680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.568287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10769.592252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10769.669305] Lustre: lustre-MDT0000: new disk, initializing [10769.773511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10769.797883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10774.116424] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10785.610435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10785.923371] Lustre: 347227:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10786.092678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10786.116090] Lustre: Skipped 1 previous similar message [10786.349779] Lustre: lustre-MDT0001: new disk, initializing [10786.527204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10786.543506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10791.916415] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10802.589983] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10808.834252] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10815.955593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10816.250795] Lustre: lustre-OST0000: new disk, initializing [10816.255407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10816.261749] Lustre: 349198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10817.911034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10817.926037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10818.038355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10823.243892] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10832.842929] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10838.489504] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10838.782894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10844.281864] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10844.603959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10848.743416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10848.749200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10848.749596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10848.756336] Lustre: Skipped 1 previous similar message [10848.761145] Lustre: Skipped 3 previous similar messages [10852.914776] Lustre: server umount lustre-OST0000 complete [10858.465510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10858.480064] LustreError: Skipped 1 previous similar message [10858.498702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10858.519225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10858.529964] Lustre: Skipped 1 previous similar message [10863.684758] Lustre: server umount lustre-MDT0000 complete [10864.096151] LustreError: 347240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10864.132388] LustreError: 347240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10868.011619] LustreError: 347219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495762 with bad export cookie 2901074163579003582 [10868.017120] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10868.021333] LustreError: 347219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10868.447875] Lustre: server umount lustre-MDT0001 complete [10877.865441] XFS (loop0): Unmounting Filesystem [10883.203875] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 13:42:55 (1782495775) [10891.806934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10892.512840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10892.522737] Lustre: Skipped 2 previous similar messages [10898.556217] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10908.636100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.990108] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10920.879496] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10926.562580] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10933.345910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10940.008125] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [10947.413876] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10958.306606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.312654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10958.330339] Lustre: Skipped 1 previous similar message [10958.338844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10958.361258] Lustre: Skipped 1 previous similar message [10962.754431] Lustre: server umount lustre-OST0000 complete [10968.559626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10968.581233] Lustre: Skipped 1 previous similar message [10973.922760] Lustre: server umount lustre-MDT0000 complete [10978.786554] LustreError: 351759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10978.841763] LustreError: 351759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10979.075968] LustreError: 351740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495873 with bad export cookie 2901074163579004464 [10979.096153] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10979.103365] LustreError: 351740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10979.669933] Lustre: server umount lustre-MDT0001 complete [10992.488954] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [10995.631324] Key type lgssc unregistered [10996.024749] LNet: 355387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10996.036936] LNetError: 355387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10996.057831] LNet: Removed LNI 192.168.204.154@tcp [10997.017840] Key type .llcrypt unregistered [10997.023449] Key type ._llcrypt unregistered [11029.165862] Key type ._llcrypt registered [11029.167925] Key type .llcrypt registered [11029.273586] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [11029.800993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11029.843984] alg: No test for adler32 (adler32-zlib) [11031.158296] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [11031.382777] LNet: Added LNI 192.168.204.154@tcp [8/256/0/180] [11033.119258] Key type lgssc registered [11034.508350] Lustre: Echo OBD driver; http://www.lustre.org/ [11042.788592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11042.814723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11044.312879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11049.347535] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11053.549305] LustreError: 356953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11053.563675] LustreError: 356953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11058.661269] LustreError: 356952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.330689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11059.774690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11064.747112] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11073.691751] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11081.082642] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11089.820747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11090.344282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11096.198432] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11104.198255] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11124.199636] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11124.210753] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11124.218199] Lustre: Skipped 1 previous similar message [11126.522465] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [11128.764065] Lustre: server umount lustre-MDT0001 complete [11129.328066] LustreError: 356953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.597205] LustreError: 356947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.642645] LustreError: 356947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11136.725062] LustreError: 358798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11136.746295] LustreError: 358798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11142.803699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11142.949606] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11142.955927] Lustre: Found index 1 for lustre-MDT0001, updating log [11143.029860] Lustre: 356941:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11143.234696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11148.274340] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11149.312199] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11150.858426] LustreError: 360489:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11150.879595] LustreError: 360489:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11150.905205] LustreError: 356944:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11151.003076] LustreError: 360497:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -17 [11151.025229] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11151.025229] [11455.466326] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11455.472323] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11455.508108] Lustre: Skipped 1 previous similar message [11455.509617] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11459.331695] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.54@tcp (stopping) [11459.338751] Lustre: Skipped 1 previous similar message [11460.544714] Lustre: server umount lustre-MDT0001 complete [11460.579294] LustreError: 357680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11460.588342] LustreError: 357680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11472.022643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11472.184126] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11472.199414] Lustre: Found index 1 for lustre-MDT0001, updating log [11472.275248] Lustre: 356942:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11472.680766] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11477.426136] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11478.011666] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11478.013624] LustreError: 368419:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11478.046055] LustreError: 368419:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11478.056420] LustreError: 356944:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11478.064547] LustreError: 356944:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11478.127825] LustreError: 368428:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11478.133653] LustreError: 368428:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11478.142503] LustreError: 368428:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -17 [11478.151195] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.154@tcp [11478.151195] [11785.187873] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11785.190772] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11785.208799] Lustre: Skipped 1 previous similar message [11785.221595] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11785.239834] Lustre: Skipped 1 previous similar message [11790.253763] Lustre: server umount lustre-MDT0001 complete [11790.309466] LustreError: 361046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11790.341244] LustreError: 361046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [11801.574468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.734827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11801.741110] Lustre: Found index 1 for lustre-MDT0001, updating log [11801.769838] Lustre: 356942:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11802.092697] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11806.316758] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [11807.221782] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11809.777715] LustreError: 377130:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11809.788903] LustreError: 377130:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11809.804475] LustreError: 356944:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11809.808452] LustreError: 356944:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11809.846618] LustreError: 377136:0:(lod_lov.c:212:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [11809.858827] LustreError: 377136:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.154@tcp: cfg command failed: rc = -114 [11809.867842] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [11809.867842] [12129.765602] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12129.769646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12129.776957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12129.776966] Lustre: Skipped 1 previous similar message [12129.781393] Lustre: Skipped 2 previous similar messages [12133.741412] Lustre: server umount lustre-MDT0000 complete [12134.881929] LustreError: 358798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12134.902816] LustreError: 358798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [12137.454925] LustreError: 356933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782497032 with bad export cookie 5192455755339968077 [12137.471733] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12137.473764] LustreError: 356933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12140.001140] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12140.023820] Lustre: Skipped 1 previous similar message [12143.892923] Lustre: server umount lustre-MDT0001 complete [12148.707695] Lustre: server umount lustre-OST0000 complete [12159.509665] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [12170.828411] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12214.153864] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 14:05:06 (1782497106) [12222.656558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12223.077407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12223.132181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12223.254834] Lustre: lustre-MDT0000: new disk, initializing [12223.404391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12223.441108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12229.333814] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12242.168517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12242.458616] Lustre: 387665:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12242.559789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12242.575765] Lustre: Skipped 1 previous similar message [12242.690208] Lustre: lustre-MDT0001: new disk, initializing [12242.793577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12242.839650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12242.856110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12248.458042] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12259.215275] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12265.694178] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12273.768830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12274.147612] Lustre: lustre-OST0000: new disk, initializing [12274.165345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12274.173445] Lustre: 389634:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12274.308543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12281.741905] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_default_debug -1 all [12283.486562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12283.508123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12283.627819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12291.867599] Lustre: DEBUG MARKER: oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12300.773795] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12300.786661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12300.805031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12300.811725] Lustre: Skipped 4 previous similar messages [12302.259410] Lustre: server umount lustre-MDT0000 complete [12304.357148] LustreError: 387677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12304.381551] LustreError: 387677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12306.197347] LustreError: 387658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782497200 with bad export cookie 5192455755339970030 [12306.210875] LustreError: MGC192.168.204.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12306.211640] LustreError: 387658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12306.216043] LustreError: 388471:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12306.654766] Lustre: server umount lustre-MDT0001 complete [12321.489516] Lustre: server umount lustre-OST0000 complete [12353.397272] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing set_hostid [12366.776948] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing load_modules_local [12403.472791] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12151 sec ========= 14:08:15 (1782497295) [12405.368054] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:08:17 (1782497297) === [12409.081460] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:08:21 (1782497301) === [12457.562372] Lustre: DEBUG MARKER: oleg454-server.virtnet: executing unload_modules_local [12461.499674] Key type lgssc unregistered [12461.903212] LNet: 396004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12461.915989] LNetError: 396004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12461.931430] LNet: Removed LNI 192.168.204.154@tcp [12462.781057] Key type .llcrypt unregistered [12462.786876] Key type ._llcrypt unregistered