[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506884568 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001024] APIC: Switch to symmetric I/O mode setup [ 0.003287] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007820] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013085] *** VALIDATE selinux *** [ 0.021788] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026608] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] *** VALIDATE tmpfs *** [ 0.031439] *** VALIDATE proc *** [ 0.032206] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041278] debug: unmapping init [mem 0xffffffff91a59000-0xffffffff91a60fff] [ 0.044183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045746] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052016] ... event mask: 000000070000000f [ 0.053314] rcu: Hierarchical SRCU implementation. [ 0.055380] smp: Bringing up secondary CPUs ... [ 0.056573] x86: Booting SMP configuration: [ 0.057030] .... node #0, CPUs: #1 #2 #3 [ 0.060510] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.086057] node 0 deferred pages initialised in 21ms [ 0.090011] devtmpfs: initialized [ 0.091045] x86/mm: Memory block size: 128MB [ 0.093556] gcov: version magic: 0x41383552 [ 0.095064] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.096061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.097236] pinctrl core: initialized pinctrl subsystem [ 0.098178] [ 0.098711] ************************************************************* [ 0.099011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.100015] ** ** [ 0.101012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.102012] ** ** [ 0.103015] ** This means that this kernel is built to expose internal ** [ 0.104013] ** IOMMU data structures, which may compromise security on ** [ 0.105017] ** your system. ** [ 0.106015] ** ** [ 0.107012] ** If you see this message and you are not debugging the ** [ 0.108011] ** kernel, report this immediately to your vendor! ** [ 0.109013] ** ** [ 0.110014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.111012] ************************************************************* [ 0.112582] NET: Registered protocol family 16 [ 0.113352] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.114066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.115051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.117030] cpuidle: using governor menu [ 0.119231] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.123000] PCI: Using configuration type 1 for base access [ 0.126157] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.137109] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.140064] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.144169] cryptd: max_cpu_qlen set to 1000 [ 0.146316] ACPI: Added _OSI(Module Device) [ 0.147017] ACPI: Added _OSI(Processor Device) [ 0.148016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.149014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.153212] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.156131] ACPI: Interpreter enabled [ 0.157047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.158009] ACPI: Using IOAPIC for interrupt routing [ 0.159127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.160411] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.171223] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.173047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.175026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.179109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.185000] acpiphp: Slot [2] registered [ 0.185000] acpiphp: Slot [5] registered [ 0.187222] acpiphp: Slot [6] registered [ 0.189140] acpiphp: Slot [7] registered [ 0.191192] acpiphp: Slot [8] registered [ 0.192215] acpiphp: Slot [9] registered [ 0.194144] acpiphp: Slot [10] registered [ 0.196192] acpiphp: Slot [3] registered [ 0.197119] acpiphp: Slot [4] registered [ 0.199190] acpiphp: Slot [11] registered [ 0.201134] acpiphp: Slot [12] registered [ 0.202166] acpiphp: Slot [13] registered [ 0.204123] acpiphp: Slot [14] registered [ 0.205105] acpiphp: Slot [15] registered [ 0.206087] acpiphp: Slot [16] registered [ 0.207174] acpiphp: Slot [17] registered [ 0.209136] acpiphp: Slot [18] registered [ 0.210129] acpiphp: Slot [19] registered [ 0.211099] acpiphp: Slot [20] registered [ 0.213124] acpiphp: Slot [21] registered [ 0.214094] acpiphp: Slot [22] registered [ 0.215081] acpiphp: Slot [23] registered [ 0.216011] acpiphp: Slot [24] registered [ 0.216903] acpiphp: Slot [25] registered [ 0.218121] acpiphp: Slot [26] registered [ 0.219115] acpiphp: Slot [27] registered [ 0.221111] acpiphp: Slot [28] registered [ 0.222124] acpiphp: Slot [29] registered [ 0.223095] acpiphp: Slot [30] registered [ 0.224072] acpiphp: Slot [31] registered [ 0.225031] PCI host bridge to bus 0000:00 [ 0.225884] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.227025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.229069] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.232038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.235030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.237026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.238202] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.242340] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.245655] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.256459] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.261108] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.263020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.265022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.267022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.269095] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.271285] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.273043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.274793] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.279020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.294020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.298022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.304895] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.312023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.320027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.335023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.347976] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.357085] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.366022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.383022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.394812] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.404020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.412021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.427026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.437489] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.446027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.454037] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.468027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.477071] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.483021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.489026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.500025] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.508497] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.514024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.520023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.535025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.545783] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.547759] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.550578] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.553374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.555154] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.558201] iommu: Default domain type: Passthrough [ 0.560667] SCSI subsystem initialized [ 0.563172] ACPI: bus type USB registered [ 0.565117] usbcore: registered new interface driver usbfs [ 0.567087] usbcore: registered new interface driver hub [ 0.569098] usbcore: registered new device driver usb [ 0.571192] pps_core: LinuxPPS API ver. 1 registered [ 0.572008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.574079] PTP clock support registered [ 0.576058] EDAC MC: Ver: 3.0.0 [ 0.577343] PCI: Using ACPI for IRQ routing [ 0.580342] NetLabel: Initializing [ 0.582023] NetLabel: domain hash size = 128 [ 0.583013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.585088] NetLabel: unlabeled traffic allowed by default [ 0.587176] vgaarb: loaded [ 0.588189] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.590019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.598322] clocksource: Switched to clocksource kvm-clock [ 0.709697] VFS: Disk quotas dquot_6.6.0 [ 0.712390] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.716485] *** VALIDATE ramfs *** [ 0.718018] *** VALIDATE hugetlbfs *** [ 0.720727] pnp: PnP ACPI init [ 0.724432] pnp: PnP ACPI: found 6 devices [ 0.744991] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.748635] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.750954] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.753399] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.755728] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.758339] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.761525] NET: Registered protocol family 2 [ 0.764294] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.769167] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.773563] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.778594] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.784736] TCP: Hash tables configured (established 65536 bind 65536) [ 0.787717] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.792496] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.795823] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.799500] NET: Registered protocol family 1 [ 0.802620] RPC: Registered named UNIX socket transport module. [ 0.805103] RPC: Registered udp transport module. [ 0.807719] RPC: Registered tcp transport module. [ 0.810542] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.814301] NET: Registered protocol family 44 [ 0.816389] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.818594] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.820688] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.823128] PCI: CLS 0 bytes, default 64 [ 0.825223] Unpacking initramfs... [ 2.309864] debug: unmapping init [mem 0xffff939bbcc54000-0xffff939bbffbffff] [ 2.316367] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.318522] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.322544] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.826546] Initialise system trusted keyrings [ 2.828063] Key type blacklist registered [ 2.829677] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.838523] zbud: loaded [ 2.841400] *** VALIDATE nfs *** [ 2.842460] *** VALIDATE nfs4 *** [ 2.843812] pstore: using deflate compression [ 2.846773] Platform Keyring initialized [ 2.946978] NET: Registered protocol family 38 [ 2.949212] Key type asymmetric registered [ 2.950671] Asymmetric key parser 'x509' registered [ 2.952500] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.955460] io scheduler mq-deadline registered [ 2.957358] io scheduler kyber registered [ 2.959308] io scheduler bfq registered [ 2.961499] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.964245] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.968031] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.971434] ACPI: Power Button [PWRF] [ 2.977065] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.984239] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.003653] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.013943] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.036618] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.066375] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.093956] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.099939] Non-volatile memory driver v1.3 [ 3.101691] Linux agpgart interface v0.103 [ 3.129609] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.131604] vda: detected capacity change from 0 to 76677120 [ 3.143705] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.145775] vdb: detected capacity change from 0 to 1073741824 [ 3.158620] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.161943] vdc: detected capacity change from 0 to 2621440000 [ 3.175337] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.177538] vdd: detected capacity change from 0 to 2621440000 [ 3.189316] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.192451] vde: detected capacity change from 0 to 4294967296 [ 3.205447] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.208144] vdf: detected capacity change from 0 to 4294967296 [ 3.213729] libphy: Fixed MDIO Bus: probed [ 3.233080] usbcore: registered new interface driver usbserial_generic [ 3.234838] usbserial: USB Serial support registered for generic [ 3.237054] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.242936] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.245386] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.248396] mousedev: PS/2 mouse device common for all mice [ 3.251963] rtc_cmos 00:05: RTC can wake from S4 [ 3.254494] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.258614] rtc_cmos 00:05: registered as rtc0 [ 3.261057] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.261533] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.266574] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.266714] intel_pstate: CPU model not supported [ 3.271401] hid: raw HID events driver (C) Jiri Kosina [ 3.272959] usbcore: registered new interface driver usbhid [ 3.274422] usbhid: USB HID core driver [ 3.275497] drop_monitor: Initializing network drop monitor service [ 3.277428] Initializing XFRM netlink socket [ 3.279303] NET: Registered protocol family 10 [ 3.282252] Segment Routing with IPv6 [ 3.283675] NET: Registered protocol family 17 [ 3.285481] mpls_gso: MPLS GSO support [ 3.293226] RAS: Correctable Errors collector initialized. [ 3.295579] AVX version of gcm_enc/dec engaged. [ 3.297537] AES CTR mode by8 optimization enabled [ 3.383139] sched_clock: Marking stable (3383099727, 0)->(4341552581, -958452854) [ 3.387235] registered taskstats version 1 [ 3.391960] Loading compiled-in X.509 certificates [ 3.394402] zswap: loaded using pool lzo/zbud [ 3.419943] Key type big_key registered [ 3.433442] Key type encrypted registered [ 3.435681] ima: No TPM chip found, activating TPM-bypass! [ 3.438126] ima: Allocated hash algorithm: sha1 [ 3.439576] ima: No architecture policies found [ 3.440931] evm: Initialising EVM extended attributes: [ 3.442499] evm: security.selinux [ 3.443550] evm: security.ima [ 3.444509] evm: security.capability [ 3.445574] evm: HMAC attrs: 0x1 [ 3.448178] rtc_cmos 00:05: setting system clock to 2026-09-03 12:51:28 UTC (1788439888) [ 3.453435] debug: unmapping init [mem 0xffffffff92a03000-0xffffffff92bfffff] [ 3.455633] debug: unmapping init [mem 0xffffffff91782000-0xffffffff91a58fff] [ 3.463622] Write protecting the kernel read-only data: 28672k [ 3.466494] debug: unmapping init [mem 0xffffffff8fe03000-0xffffffff8fffffff] [ 3.468945] debug: unmapping init [mem 0xffffffff90714000-0xffffffff907fffff] [ 3.499541] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.506612] systemd[1]: Detected virtualization kvm. [ 3.507950] systemd[1]: Detected architecture x86-64. [ 3.509412] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.532910] systemd[1]: No hostname configured. [ 3.534587] systemd[1]: Set hostname to . [ 3.536056] random: systemd: uninitialized urandom read (16 bytes read) [ 3.537745] systemd[1]: Initializing machine ID from random generator. [ 3.638908] random: systemd: uninitialized urandom read (16 bytes read) [ 3.641833] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.647706] random: systemd: uninitialized urandom read (16 bytes read) [ 3.650942] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.655262] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.249593] device-mapper: uevent: version 1.0.3 [ 4.252118] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hard[ 5.000098] random: fast init done ware RNG Entropy Gatherer Daemon. [ 5.024816] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 5.085141] scsi host0: ata_piix [ 5.103372] scsi host1: ata_piix [ 5.105338] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.107947] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.505668] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.950958] random: crng init done [ 9.952461] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.384129] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.556858] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.811872] SELinux: Disabled at runtime. [ 11.879382] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.888645] systemd[1]: Detected virtualization kvm. [ 11.890785] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.353911] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.357855] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.361697] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.365353] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.368462] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.376700] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.380134] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.407926] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 12.730467] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.019532] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.030916] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.165635] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.176838] EDAC sbridge: Ver: 1.1.2 [ 14.816199] Key type dns_resolver registered [ 15.137452] NFS: Registering the id_resolver key type [ 15.140206] Key type id_resolver registered [ 15.142444] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg121-server login: [ 41.161415] libcfs: loading out-of-tree module taints kernel. [ 41.204747] Key type ._llcrypt registered [ 41.206946] Key type .llcrypt registered [ 41.298173] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 60.853402] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 62.066514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.075952] alg: No test for adler32 (adler32-zlib) [ 63.604784] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 64.360379] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 66.015355] Key type lgssc registered [ 67.495618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.462752] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 93.425339] hrtimer: interrupt took 6721070 ns [ 133.181661] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 147.654072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 147.684772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.116566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 149.147745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 149.356053] Lustre: lustre-MDT0000: new disk, initializing [ 149.485369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.515290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.571302] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.137560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.262532] Lustre: 6508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 171.365309] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 171.379497] Lustre: Skipped 1 previous similar message [ 171.551545] Lustre: lustre-MDT0001: new disk, initializing [ 171.627828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.700160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.716688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.887388] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.599411] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.079318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.398592] Lustre: lustre-OST0000: new disk, initializing [ 191.402828] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 191.409996] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.496690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.727774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 194.757313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 194.909556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.615760] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.602823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.780905] Lustre: lustre-OST0001: new disk, initializing [ 215.786790] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 215.796556] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.922401] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.328449] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.787757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.811507] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.896308] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 235.494863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.521389] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 249.095978] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.072248] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing yml_node [ 259.030536] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 261.871976] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 264.614463] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 266.682236] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:55:51 EDT 2026 [ 284.559842] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 286.509990] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 304.612863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 304.629439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.650770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.648751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.652572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.676275] Lustre: Skipped 1 previous similar message [ 309.799331] Lustre: server umount lustre-MDT0000 complete [ 313.347021] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440198 with bad export cookie 10397177709848911126 [ 313.347846] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.372902] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 315.871589] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.873104] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.873318] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.873323] Lustre: Skipped 3 previous similar messages [ 315.884153] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 315.893032] Lustre: Skipped 2 previous similar messages [ 318.880149] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 318.880642] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 318.923788] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 318.943237] Lustre: Skipped 2 previous similar messages [ 319.999876] Lustre: server umount lustre-MDT0001 complete [ 324.266669] Lustre: server umount lustre-OST0000 complete [ 328.221234] Lustre: server umount lustre-OST0001 complete [ 335.310509] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 346.346919] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 390.312200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.854240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 390.910857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 391.087203] Lustre: lustre-MDT0000: new disk, initializing [ 391.286326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.318938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 396.508177] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 409.360334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.489582] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 409.541860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 409.544449] Lustre: Skipped 1 previous similar message [ 409.667050] Lustre: lustre-MDT0001: new disk, initializing [ 409.770656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 409.819822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 409.837337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 414.481236] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 425.602029] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 432.641105] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 440.397280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.703769] Lustre: lustre-OST0000: new disk, initializing [ 440.711672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 440.716809] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 440.816936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.696551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 442.716852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 442.872816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.668449] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 459.732882] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 464.822551] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 465.006164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.414940] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 470.713735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 473.579210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.590870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.592201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.593674] Lustre: Skipped 1 previous similar message [ 473.610791] LustreError: Skipped 1 previous similar message [ 479.516861] Lustre: server umount lustre-OST0000 complete [ 486.881844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 486.887501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.912573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.923172] Lustre: Skipped 3 previous similar messages [ 490.375317] Lustre: server umount lustre-MDT0000 complete [ 494.052850] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.079963] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 494.328662] LustreError: 17043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440379 with bad export cookie 10397177709848912659 [ 494.331193] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.343019] LustreError: 17043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 494.783712] Lustre: server umount lustre-MDT0001 complete [ 501.026824] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 08:59:45 (1788440385) [ 507.846394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.385076] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.467832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.512835] LustreError: 19309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.576723] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 523.564751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.747945] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.767504] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 528.134907] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 536.884902] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 543.166792] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 551.298967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 560.137978] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 570.000091] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 573.882572] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 580.579154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.582756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.610929] Lustre: Skipped 1 previous similar message [ 580.621567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 580.628728] Lustre: Skipped 2 previous similar messages [ 585.608654] Lustre: server umount lustre-MDT0000 complete [ 588.778131] LustreError: 19309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.788276] LustreError: 19309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 589.215707] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440474 with bad export cookie 10397177709848913534 [ 589.217462] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 589.236022] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 589.738833] Lustre: server umount lustre-MDT0001 complete [ 604.216866] Lustre: server umount lustre-OST0000 complete [ 627.597344] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 638.424635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.921185] LustreError: 23704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.986975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.993415] Lustre: Skipped 2 previous similar messages [ 644.381382] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 653.389957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.072256] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 660.790237] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 667.437335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 674.196804] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 682.822650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 682.988569] Lustre: lustre-OST0001: new disk, initializing [ 682.992059] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 682.996804] Lustre: 26023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 684.893727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 684.908667] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 685.051390] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 689.318405] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 699.088996] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 702.161812] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 715.232666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 715.244674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.277361] Lustre: Skipped 2 previous similar messages [ 715.289692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 715.296536] Lustre: Skipped 3 previous similar messages [ 719.990694] Lustre: server umount lustre-MDT0000 complete [ 720.889990] LustreError: 23699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 720.915128] LustreError: 23699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 723.831050] LustreError: 23686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440608 with bad export cookie 10397177709848914388 [ 723.837040] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.846582] LustreError: 23686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 724.194503] Lustre: server umount lustre-MDT0001 complete [ 737.195863] Lustre: server umount lustre-OST0000 complete [ 751.026771] Lustre: server umount lustre-OST0001 complete [ 767.706711] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 778.674922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.374624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.400337] Lustre: Skipped 3 previous similar messages [ 783.465461] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 792.975651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.949432] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 800.801753] Lustre: 30191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 807.631490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 814.876311] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 823.214961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 830.295806] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 838.422722] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 842.313808] Lustre: 32071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 849.888778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 849.898148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.910428] Lustre: Skipped 3 previous similar messages [ 849.915804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 849.918587] Lustre: Skipped 3 previous similar messages [ 855.246870] Lustre: server umount lustre-MDT0000 complete [ 855.537466] LustreError: 29038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.568042] LustreError: 29038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 859.129884] LustreError: 29024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440744 with bad export cookie 10397177709848915676 [ 859.130335] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 859.158677] LustreError: 29024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 859.695277] Lustre: server umount lustre-MDT0001 complete [ 873.260921] Lustre: server umount lustre-OST0000 complete [ 876.515149] Lustre: 3639:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440745/real 1788440745] req@ffff939c3876b800 x1875315208257792/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440761 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 877.166445] Lustre: server umount lustre-OST0001 complete [ 886.618271] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 09:06:11 (1788440771) [ 914.607635] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 926.647538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.791245] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 943.029905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.445844] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 951.050936] Lustre: 36022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 957.964334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 964.255241] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 972.423885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 978.348736] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 985.879871] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 989.827479] Lustre: 37900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 994.787955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 994.792789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.800215] Lustre: Skipped 4 previous similar messages [ 994.803883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.809093] Lustre: Skipped 4 previous similar messages [ 1001.128563] Lustre: server umount lustre-MDT0000 complete [ 1004.605389] LustreError: 36024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440889 with bad export cookie 10397177709848916789 [ 1004.607338] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1004.623215] LustreError: 36024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1004.959608] Lustre: server umount lustre-MDT0001 complete [ 1018.179254] Lustre: server umount lustre-OST0000 complete [ 1032.382698] Lustre: server umount lustre-OST0001 complete [ 1044.437792] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 09:08:48 (1788440928) [ 1065.099804] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1075.846546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.162313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.169212] Lustre: Skipped 7 previous similar messages [ 1081.448339] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1089.941631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.625559] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1100.165865] Lustre: 41199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1107.054931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.677443] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1115.755515] LustreError: 41552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.784222] LustreError: 41552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 1123.195597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1129.716773] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1138.521866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1142.549395] Lustre: 43075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1143.646472] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1143.651593] Lustre: Skipped 1 previous similar message [ 1146.850592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1146.854626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.882118] Lustre: Skipped 3 previous similar messages [ 1152.596306] Lustre: server umount lustre-MDT0000 complete [ 1156.868329] LustreError: 40028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441041 with bad export cookie 10397177709848917895 [ 1156.892074] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.356429] Lustre: server umount lustre-MDT0001 complete [ 1172.607040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.918495] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1191.096918] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1197.063759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.163723] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1203.189906] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1203.194477] Lustre: Skipped 1 previous similar message [ 1209.613304] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1211.835969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1218.513563] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1220.183558] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1228.260585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1232.773296] Lustre: server umount lustre-MDT0000 complete [ 1236.076940] LustreError: 40030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441121 with bad export cookie 10397177709848918994 [ 1236.082924] LustreError: 40030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1236.084556] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1236.505568] Lustre: server umount lustre-MDT0001 complete [ 1250.662692] Lustre: server umount lustre-OST0000 complete [ 1264.196562] Lustre: server umount lustre-OST0001 complete [ 1273.209514] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 09:12:37 (1788441157) [ 1293.743445] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1306.074937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.268790] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1320.626620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.709691] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1329.433266] Lustre: 48838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1337.388082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1347.299941] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1358.938311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1366.194589] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1374.323396] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1378.817835] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1395.179959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1395.188222] Lustre: Skipped 21 previous similar messages [ 1399.077644] Lustre: server umount lustre-MDT0000 complete [ 1402.423871] LustreError: 47665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441287 with bad export cookie 10397177709848919820 [ 1402.425745] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1402.441483] LustreError: 47665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1402.852743] Lustre: server umount lustre-MDT0001 complete [ 1416.229751] Lustre: server umount lustre-OST0000 complete [ 1430.538827] Lustre: server umount lustre-OST0001 complete [ 1441.130674] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 09:15:25 (1788441325) [ 1444.093166] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1446.098809] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 09:15:30 (1788441330) [ 1457.330512] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 1467.001488] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1491.896762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.122487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1492.192763] Lustre: lustre-MDT0000: new disk, initializing [ 1492.343641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1497.118586] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1507.114892] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1513.971964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1514.262168] Lustre: lustre-OST0000: new disk, initializing [ 1514.266965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1514.274952] Lustre: Skipped 1 previous similar message [ 1514.283652] Lustre: 54916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1516.223497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1516.229276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1516.270853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1520.830527] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1530.906967] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1533.135812] Lustre: DEBUG MARKER: create test files [ 1567.715195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1567.720407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1567.726808] Lustre: Skipped 13 previous similar messages [ 1573.800347] Lustre: server umount lustre-OST0000 complete [ 1578.911626] Lustre: server umount lustre-MDT0000 complete [ 1593.970363] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1597.858285] Key type lgssc unregistered [ 1598.410290] LNet: 56547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.425560] LNetError: 56547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.461893] LNet: Removed LNI 192.168.201.121@tcp [ 1601.069165] Key type .llcrypt unregistered [ 1601.076492] Key type ._llcrypt unregistered [ 1636.496629] Key type ._llcrypt registered [ 1636.498980] Key type .llcrypt registered [ 1636.631388] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1638.161513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1638.191161] alg: No test for adler32 (adler32-zlib) [ 1639.174177] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1639.430461] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1641.167271] Key type lgssc registered [ 1642.453193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1650.241428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1650.266495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.872047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.030949] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1666.440652] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1673.717677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1674.104032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.156485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1676.313272] Lustre: 59285:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1676.321442] Lustre: 59285:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1676.330872] Lustre: 59285:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1676.340468] Lustre: 59285:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1676.355619] Lustre: 59285:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1676.359031] Lustre: 59285:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1680.795428] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1689.909737] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1695.597735] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1702.532514] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1712.100162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1712.113672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.416833] Lustre: server umount lustre-OST0000 complete [ 1716.973503] Lustre: server umount lustre-MDT0000 complete [ 1729.834353] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1733.309104] Key type lgssc unregistered [ 1733.719080] LNet: 60893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.727476] LNetError: 60893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.758621] LNet: Removed LNI 192.168.201.121@tcp [ 1735.921613] Key type .llcrypt unregistered [ 1735.927912] Key type ._llcrypt unregistered [ 1768.301703] Key type ._llcrypt registered [ 1768.304800] Key type .llcrypt registered [ 1768.395376] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1769.754120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1769.778077] alg: No test for adler32 (adler32-zlib) [ 1770.892422] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1771.093200] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1772.735216] Key type lgssc registered [ 1774.320372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1782.054923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1782.099550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.611244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.470385] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1797.014695] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1804.177026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1804.405447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.490813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1808.542814] Lustre: 63735:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1808.546689] Lustre: 63735:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1808.551175] Lustre: 63735:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1808.556286] Lustre: 63735:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1808.561256] Lustre: 63735:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1808.565316] Lustre: 63735:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1811.299136] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1820.506546] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1826.644829] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1839.586419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.595056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.639276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1842.968777] Lustre: server umount lustre-OST0000 complete [ 1848.693598] Lustre: server umount lustre-MDT0000 complete [ 1861.533614] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 1864.319604] Key type lgssc unregistered [ 1864.566698] LNet: 65293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.586266] LNetError: 65293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.606557] LNet: Removed LNI 192.168.201.121@tcp [ 1865.413468] Key type .llcrypt unregistered [ 1865.419894] Key type ._llcrypt unregistered [ 1885.606662] Key type ._llcrypt registered [ 1885.615884] Key type .llcrypt registered [ 1885.756195] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 1903.835811] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 1904.956022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1905.073757] alg: No test for adler32 (adler32-zlib) [ 1906.240382] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1906.568697] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 1908.247237] Key type lgssc registered [ 1909.442362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1953.668352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1953.683794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.044234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1955.060860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1955.205384] Lustre: lustre-MDT0000: new disk, initializing [ 1955.341769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.366570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1961.310219] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1973.937644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.116455] Lustre: 68446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1974.242451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1974.250734] Lustre: Skipped 1 previous similar message [ 1974.389456] Lustre: lustre-MDT0001: new disk, initializing [ 1974.544844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1974.598475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1974.612975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1980.108178] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 1989.940671] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1996.986094] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2004.390673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.736187] Lustre: lustre-OST0000: new disk, initializing [ 2004.741529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2004.755670] Lustre: 70415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2004.855182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.207710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2011.222613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2011.233977] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2011.375676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2021.041363] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2026.264210] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2026.487238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2031.030436] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2031.194772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2037.218049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2037.235967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.239420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.250273] Lustre: Skipped 1 previous similar message [ 2037.263502] Lustre: Skipped 1 previous similar message [ 2039.374341] Lustre: server umount lustre-OST0000 complete [ 2046.433064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2046.451379] LustreError: Skipped 1 previous similar message [ 2046.465952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.475870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2050.068359] Lustre: server umount lustre-MDT0000 complete [ 2054.101513] LustreError: 68436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441939 with bad export cookie 17945310895386290126 [ 2054.107216] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2054.108211] LustreError: 68436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2054.402195] Lustre: server umount lustre-MDT0001 complete [ 2064.523490] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 09:25:49 (1788441949) [ 2076.381089] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2118.835543] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2129.165528] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2168.413042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.764620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2168.812624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2168.902378] Lustre: lustre-MDT0000: new disk, initializing [ 2169.054336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.095792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2173.515611] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2185.282301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.413062] Lustre: 76435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2185.470601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2185.474480] Lustre: Skipped 1 previous similar message [ 2185.525604] Lustre: lustre-MDT0001: new disk, initializing [ 2185.616586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2185.670764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2185.692139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2190.700401] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2202.592258] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2210.668886] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2219.063467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2219.311596] Lustre: lustre-OST0000: new disk, initializing [ 2219.321285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2219.330394] Lustre: 78405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2219.420255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2226.242498] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2226.730293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2226.736135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2226.823500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2236.133680] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2241.559534] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2241.848503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2248.506457] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2248.834098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2252.258713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2252.264944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2252.278209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2252.278216] Lustre: Skipped 2 previous similar messages [ 2252.300026] Lustre: Skipped 2 previous similar messages [ 2257.393556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2257.405729] Lustre: Skipped 2 previous similar messages [ 2257.864582] Lustre: server umount lustre-OST0000 complete [ 2267.618524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2267.620062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.633128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2267.642472] Lustre: Skipped 1 previous similar message [ 2267.655554] Lustre: Skipped 1 previous similar message [ 2269.070371] Lustre: server umount lustre-MDT0000 complete [ 2272.747459] LustreError: 76443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2272.786848] LustreError: 76443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2273.662711] LustreError: 77230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442158 with bad export cookie 17945310895386291008 [ 2273.674424] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2273.677966] LustreError: 77230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2274.210699] Lustre: server umount lustre-MDT0001 complete [ 2284.015239] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:29:28 (1788442168) [ 2291.843156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.522333] LustreError: 80818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.632969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.401955] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2297.825355] LustreError: 80819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2302.954111] LustreError: 80818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.070123] LustreError: 80819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.414343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.919879] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2321.867857] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2329.286445] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2337.800345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2338.197688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.204953] Lustre: Skipped 1 previous similar message [ 2345.347714] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2353.547931] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2357.965943] Lustre: 83489:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting ddd3dea1-14db-4bd0-b70f-72f69251e875 at adminstrative request [ 2358.760661] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2358.783137] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2367.984897] Lustre: MGS: Client ddd3dea1-14db-4bd0-b70f-72f69251e875 (at 0@lo) reconnecting [ 2367.991745] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0xf90a8d2e57726eb2 to 0xf90a8d2e57727152 [ 2368.002068] Lustre: MGC192.168.201.121@tcp: Connection restored to 0@lo (at 0@lo) [ 2396.046496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.205922] Lustre: lustre-OST0001: new disk, initializing [ 2396.209475] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2396.213635] Lustre: 83883:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2404.122751] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2405.496437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2405.503946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2405.621685] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2415.228240] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2421.238745] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2421.243371] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.253362] LustreError: Skipped 1 previous similar message [ 2421.256899] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2421.294506] Lustre: Skipped 1 previous similar message [ 2424.754588] Lustre: server umount lustre-OST0001 complete [ 2426.347250] LustreError: 84313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.369324] LustreError: 84313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2430.434265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.451745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.657184] Lustre: server umount lustre-OST0000 complete [ 2441.731319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.738820] Lustre: Skipped 5 previous similar messages [ 2445.290615] Lustre: server umount lustre-MDT0000 complete [ 2446.821658] LustreError: 80819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.835850] LustreError: 80819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2449.270923] LustreError: 81553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442334 with bad export cookie 17945310895386292562 [ 2449.286478] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.289107] LustreError: 81553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2449.623141] Lustre: server umount lustre-MDT0001 complete [ 2458.856388] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:32:23 (1788442343) [ 2460.614712] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2462.858699] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:32:27 (1788442347) [ 2464.401569] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2466.260862] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:32:31 (1788442351) [ 2468.383192] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2470.884504] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 09:32:35 (1788442355) [ 2473.190228] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2475.442366] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:32:39 (1788442359) [ 2477.587720] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2484.632482] loop: module loaded [ 2484.664677] loop0: detected capacity change from 0 to 4294967296 [ 2484.732756] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2486.529422] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2490.365872] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2494.579354] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2523.762136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2523.816681] LustreError: 88009:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2523.830849] LustreError: 88009:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2523.836277] LustreError: 88009:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2523.842975] LustreError: 88009:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2545.999189] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 2555.477966] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 2593.530861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2593.765100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2593.794822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2593.857978] Lustre: lustre-MDT0000: new disk, initializing [ 2593.911088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.919696] Lustre: Skipped 1 previous similar message [ 2593.931983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2598.373290] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2609.195786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.338310] Lustre: 90875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2609.778051] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2613.940459] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2623.150573] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2628.992084] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2636.275886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.751236] Lustre: 92842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2641.926285] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2645.588869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2645.693606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2650.990447] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2655.863838] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2656.118261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2660.946487] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2661.252610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2666.473738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.476197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2666.484234] Lustre: Skipped 3 previous similar messages [ 2666.485241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2666.504608] LustreError: Skipped 2 previous similar messages [ 2669.861376] Lustre: server umount lustre-OST0000 complete [ 2680.146842] Lustre: server umount lustre-MDT0000 complete [ 2681.827275] LustreError: 90885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.846586] LustreError: 90885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2683.803787] LustreError: 90866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442568 with bad export cookie 17945310895386293178 [ 2683.812497] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2683.817226] LustreError: 90866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2684.071391] Lustre: server umount lustre-MDT0001 complete [ 2693.999868] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 09:36:18 (1788442578) [ 2701.838072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2706.984591] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2716.808806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.173657] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2730.498609] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2736.328775] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2743.403721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2743.727243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.737956] Lustre: Skipped 4 previous similar messages [ 2749.574542] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2757.404194] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2762.035606] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2762.290340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2767.260688] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2767.581719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2775.666370] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2775.900405] Lustre: lustre-OST0001: new disk, initializing [ 2775.907619] Lustre: Skipped 2 previous similar messages [ 2775.924669] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2775.936885] Lustre: Skipped 3 previous similar messages [ 2775.944978] Lustre: 98622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2781.828095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2781.853396] Lustre: Skipped 2 previous similar messages [ 2781.856061] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2782.042672] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2784.820601] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2797.405076] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2803.903239] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2804.311524] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2809.978459] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2810.285600] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2833.237611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2859.498567] Lustre: Failing over lustre-MDT0000 [ 2859.713865] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 2859.724235] Lustre: Skipped 3 previous similar messages [ 2860.181729] Lustre: server umount lustre-MDT0000 complete [ 2860.513797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2860.519486] LustreError: Skipped 1 previous similar message [ 2860.526557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.540194] Lustre: Skipped 3 previous similar messages [ 2860.544503] LustreError: 95253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2860.570655] LustreError: 95253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2880.991168] Lustre: 66665:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442749/real 1788442749] req@000000006c72f2f9 x1875317139722240/t0(0) o400->MGC192.168.201.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788442765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2881.017124] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2886.239931] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2886.242864] LDISKFS-fs (dm-0): recovery complete [ 2886.249559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2891.240900] LustreError: 66664:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d66555bf x1875317139731584/t0(0) o250->MGC192.168.201.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2891.748685] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2896.559499] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2896.559751] Lustre: 95248:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2896.913207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2897.003222] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2897.344101] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2897.366259] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2897.392602] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2897.724208] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2897.741937] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2897.916408] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 2898.097890] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2898.101707] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2898.462229] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2898.486724] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 2 previous similar messages [ 2898.812787] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2898.826291] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2899.151674] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2899.182954] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2900.598798] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2900.629364] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 5 previous similar messages [ 2900.960526] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2900.998720] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2901.337145] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2901.364099] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2904.655604] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2904.677047] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 11 previous similar messages [ 2905.016783] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2905.038048] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 11 previous similar messages [ 2905.400940] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2905.415550] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 11 previous similar messages [ 2912.765565] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2912.789550] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 23 previous similar messages [ 2913.166186] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2913.215547] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 23 previous similar messages [ 2913.549913] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2913.567258] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 23 previous similar messages [ 2928.889813] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2928.907464] Lustre: 101477:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 49 previous similar messages [ 2929.233024] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2929.251205] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 2929.575120] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2929.589041] LustreError: 101477:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 2956.015639] Lustre: 101477:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2956.028596] Lustre: 101477:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2956.046962] Lustre: 101477:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client aaf272b9-9aa2-44f9-9b9b-08d7f5e78244@192.168.201.21@tcp [ 2956.066578] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2956.084164] Lustre: 101477:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@00000000122f0166 x1875317122734336/t0(8589934778) o35->aaf272b9-9aa2-44f9-9b9b-08d7f5e78244@192.168.201.21@tcp:361/0 lens 392/0 e 0 to 0 dl 1788442851 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2956.121856] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2956.132840] Lustre: lustre-MDT0000: Denying connection for new client aaf272b9-9aa2-44f9-9b9b-08d7f5e78244 (at 192.168.201.21@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2956.155962] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2956.228549] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2956.240471] Lustre: Skipped 2 previous similar messages [ 2956.262621] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2956.325028] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 2956.325727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:80 to 0x280000401:577) [ 2977.249741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2977.261357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2977.283905] Lustre: Skipped 3 previous similar messages [ 2982.271558] Lustre: server umount lustre-OST0000 complete [ 2988.006871] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2988.023032] Lustre: Skipped 8 previous similar messages [ 2989.035290] LustreError: 97075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.064156] LustreError: 97075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 2992.629122] Lustre: server umount lustre-OST0001 complete [ 3003.079976] Lustre: server umount lustre-MDT0000 complete [ 3007.048334] LustreError: 95236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442892 with bad export cookie 17945310895386344390 [ 3007.057679] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3007.072692] LustreError: 95236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3007.421289] Lustre: server umount lustre-MDT0001 complete [ 3017.453972] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:41:42 (1788442902) [ 3025.524334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.530791] Lustre: *** cfs_fail_loc=197, val=0*** [ 3041.120571] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3046.239628] LustreError: 103459:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 3051.404961] Lustre: *** cfs_fail_loc=197, val=0*** [ 3051.411043] Lustre: Skipped 65 previous similar messages [ 3071.967328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.976465] Lustre: Skipped 2 previous similar messages [ 3078.437711] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3087.477231] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3089.895207] Lustre: server umount lustre-OST0000 complete [ 3101.941451] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 09:43:06 (1788442986) [ 3125.931384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.736313] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3143.700401] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3148.568207] LustreError: 105111:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3148.582422] LustreError: 105111:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3148.595487] LustreError: 105111:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 3149.318913] Lustre: server umount lustre-MDT0000 complete [ 3187.119851] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3199.367285] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3240.318889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.628414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3240.658053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3240.751712] Lustre: lustre-MDT0000: new disk, initializing [ 3240.810202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3245.710540] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3258.901959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.035790] Lustre: 109288:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3259.430653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3264.216787] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3275.741132] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3283.459048] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3292.124979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3292.531608] Lustre: 111259:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3299.670787] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3302.144401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3311.118609] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3318.544360] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3318.933253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3326.364281] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3326.567952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3333.106142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3333.114455] LustreError: Skipped 4 previous similar messages [ 3333.114734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.132696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3333.132703] Lustre: Skipped 5 previous similar messages [ 3333.166966] Lustre: Skipped 6 previous similar messages [ 3343.327906] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3343.721977] Lustre: server umount lustre-OST0000 complete [ 3354.910886] Lustre: server umount lustre-MDT0000 complete [ 3358.689671] LustreError: 109295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3358.728796] LustreError: 109295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3359.131829] LustreError: 111258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443244 with bad export cookie 17945310895386355072 [ 3359.139948] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3359.144810] LustreError: 111258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3359.452861] Lustre: server umount lustre-MDT0001 complete [ 3368.278821] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:47:33 (1788443253) [ 3377.977927] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3380.616775] Key type lgssc unregistered [ 3380.966085] LNet: 113748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3380.973628] LNetError: 113748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3382.054506] LNet: Removed LNI 192.168.201.121@tcp [ 3382.772932] Key type .llcrypt unregistered [ 3382.775395] Key type ._llcrypt unregistered [ 3401.692667] Key type ._llcrypt registered [ 3401.698561] Key type .llcrypt registered [ 3401.841662] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3417.067897] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3418.041384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3418.067127] alg: No test for adler32 (adler32-zlib) [ 3419.236248] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3419.442257] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 3421.087165] Key type lgssc registered [ 3422.575257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3470.122267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3470.158340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.545766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3471.595299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3471.723980] Lustre: lustre-MDT0000: new disk, initializing [ 3471.861118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3471.883961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3476.275536] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3486.986185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3487.288560] Lustre: lustre-OST0000: new disk, initializing [ 3487.292980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3487.304367] Lustre: Skipped 1 previous similar message [ 3487.315808] Lustre: 117919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3487.380524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.556918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3487.569986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3487.621368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3494.854526] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3508.730029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.135555] Lustre: lustre-OST0001: new disk, initializing [ 3509.150385] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3509.174687] Lustre: 118782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3509.441934] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3518.205688] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3519.113742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3519.131900] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3519.174362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3529.189351] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.198806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.231691] Lustre: Skipped 1 previous similar message [ 3531.718865] Lustre: server umount lustre-MDT0000 complete [ 3542.507315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3549.747483] LustreError: 117107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443434 with bad export cookie 14569616881311779896 [ 3549.758544] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3550.020233] Lustre: server umount lustre-OST0000 complete [ 3554.510978] Lustre: server umount lustre-OST0001 complete [ 3565.469807] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 09:50:49 (1788443449) [ 3592.858783] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 09:51:17 (1788443477) [ 3605.512980] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3632.192404] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:51:57 (1788443517) [ 3679.841101] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 3690.709245] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 3734.474648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3734.702337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3734.737503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3734.795806] Lustre: lustre-MDT0000: new disk, initializing [ 3734.849915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3734.868828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3740.491806] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3753.232105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3753.395781] Lustre: 126542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3753.469269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3753.475272] Lustre: Skipped 1 previous similar message [ 3753.602335] Lustre: lustre-MDT0001: new disk, initializing [ 3753.772166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3753.841526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3753.854508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3759.152511] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3769.870697] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3776.997700] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3785.714970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3786.051800] Lustre: lustre-OST0000: new disk, initializing [ 3786.059991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3786.074227] Lustre: 128511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3786.210437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3792.945288] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3796.064491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3796.078795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3796.246914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3802.826344] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3808.260332] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3808.495166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3814.483900] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3814.820500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3822.050930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.063873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3822.071318] Lustre: Skipped 1 previous similar message [ 3822.072929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.083908] Lustre: Skipped 2 previous similar messages [ 3823.512890] Lustre: server umount lustre-OST0000 complete [ 3830.752469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3830.764759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3830.781805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3834.833910] Lustre: server umount lustre-MDT0000 complete [ 3838.678403] LustreError: 126534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443723 with bad export cookie 14569616881311781121 [ 3838.682994] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3838.689288] LustreError: 126534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3838.989039] Lustre: server umount lustre-MDT0001 complete [ 3850.257919] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:55:35 (1788443735) [ 3857.157144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.750174] LustreError: 130923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3857.776024] LustreError: 130923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3857.833195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.064294] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3863.017815] LustreError: 130924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3867.121735] LustreError: 130923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.775556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3876.662882] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3884.531812] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3891.645050] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3898.975785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3899.275099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3899.279389] Lustre: Skipped 1 previous similar message [ 3906.088421] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 3913.896347] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3921.862049] Lustre: *** cfs_fail_loc=159, val=0*** [ 3921.875745] Lustre: Skipped 2 previous similar messages [ 3938.532301] Lustre: lustre-MDT0000: Client 9e272f3b-78b7-4410-abb6-faf8db712794 (at 192.168.201.21@tcp) reconnecting [ 3940.440061] Lustre: *** cfs_fail_loc=159, val=0*** [ 3940.441781] Lustre: Skipped 3 previous similar messages [ 3955.928573] Lustre: lustre-MDT0000: Client 9e272f3b-78b7-4410-abb6-faf8db712794 (at 192.168.201.21@tcp) reconnecting [ 3963.202835] Lustre: server umount lustre-OST0000 complete [ 3968.996276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.004839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3969.010066] Lustre: Skipped 2 previous similar messages [ 3969.032969] Lustre: Skipped 3 previous similar messages [ 3974.115838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3974.126421] Lustre: Skipped 1 previous similar message [ 3976.290209] Lustre: server umount lustre-MDT0000 complete [ 3979.232419] LustreError: 133531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3979.234544] Lustre: 115122:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443848/real 1788443848] req@000000007836d607 x1875318726044928/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788443864 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3979.257910] LustreError: 133531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3979.318943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.926959] LustreError: 133532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443864 with bad export cookie 14569616881311782003 [ 3979.944550] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3980.332087] Lustre: server umount lustre-MDT0001 complete [ 3992.308863] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 3994.799537] Key type lgssc unregistered [ 3995.099288] LNet: 134707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3995.103510] LNetError: 134707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3996.134260] LNet: Removed LNI 192.168.201.121@tcp [ 3997.026582] Key type .llcrypt unregistered [ 3997.028628] Key type ._llcrypt unregistered [ 4023.296638] Key type ._llcrypt registered [ 4023.300410] Key type .llcrypt registered [ 4023.418965] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4024.796043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4024.885066] alg: No test for adler32 (adler32-zlib) [ 4026.006372] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4026.239387] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4027.904582] Key type lgssc registered [ 4029.402066] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4037.350889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4037.365819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.713068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.745632] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4048.870442] LustreError: 136118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4048.884972] LustreError: 136118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4051.956630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.376526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4057.242275] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4067.820558] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4075.835606] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4085.709601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.997149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.324711] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4104.248515] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4108.551064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4121.834374] Lustre: *** cfs_fail_loc=159, val=0*** [ 4138.225315] Lustre: lustre-MDT0000: Client f88eaa33-71b6-4fcd-83bf-2b100ef64307 (at 192.168.201.21@tcp) reconnecting [ 4141.681327] Lustre: *** cfs_fail_loc=159, val=0*** [ 4158.204302] Lustre: lustre-MDT0001: Client f88eaa33-71b6-4fcd-83bf-2b100ef64307 (at 192.168.201.21@tcp) reconnecting [ 4160.154182] Lustre: *** cfs_fail_loc=159, val=0*** [ 4160.156018] Lustre: Skipped 3 previous similar messages [ 4176.697234] Lustre: lustre-MDT0001: Client f88eaa33-71b6-4fcd-83bf-2b100ef64307 (at 192.168.201.21@tcp) reconnecting [ 4181.762750] Lustre: 139191:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4185.901469] Lustre: *** cfs_fail_loc=159, val=0*** [ 4185.923265] Lustre: Skipped 5 previous similar messages [ 4201.866771] Lustre: lustre-MDT0000: Client f1752a9d-0bf9-4e1b-a6c1-abec78741de6 (at 192.168.201.21@tcp) reconnecting [ 4204.180296] Lustre: *** cfs_fail_loc=159, val=0*** [ 4204.182055] Lustre: Skipped 13 previous similar messages [ 4220.660344] Lustre: lustre-MDT0000: Client f1752a9d-0bf9-4e1b-a6c1-abec78741de6 (at 192.168.201.21@tcp) reconnecting [ 4222.566802] Lustre: 139483:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4241.894859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.910760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4241.916248] Lustre: Skipped 1 previous similar message [ 4247.012928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4247.022724] Lustre: Skipped 1 previous similar message [ 4248.099286] Lustre: server umount lustre-OST0000 complete [ 4257.274358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.275195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4257.279616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4257.291443] Lustre: Skipped 2 previous similar messages [ 4259.402338] Lustre: server umount lustre-MDT0000 complete [ 4262.373941] LustreError: 139252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.389981] LustreError: 139252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4264.114471] LustreError: 136098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444149 with bad export cookie 8016041929190520619 [ 4264.124872] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4264.151074] LustreError: 136098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4264.536679] Lustre: server umount lustre-MDT0001 complete [ 4277.077634] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4280.211604] Key type lgssc unregistered [ 4280.640677] LNet: 140561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4280.645416] LNetError: 140561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4280.676715] LNet: Removed LNI 192.168.201.121@tcp [ 4281.672421] Key type .llcrypt unregistered [ 4281.677079] Key type ._llcrypt unregistered [ 4311.012744] Key type ._llcrypt registered [ 4311.016919] Key type .llcrypt registered [ 4311.243302] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4312.921923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4312.959365] alg: No test for adler32 (adler32-zlib) [ 4314.223660] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4314.387869] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4316.063525] Key type lgssc registered [ 4317.563773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4326.435852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4326.458958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.989897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.587658] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4337.130403] LustreError: 141974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4337.153267] LustreError: 141974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4342.247450] LustreError: 141975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.308125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4343.555177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4348.171525] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4357.814333] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4365.696586] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4373.429203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4373.690520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.504392] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4388.047857] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4388.553350] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4397.240125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4404.138424] Lustre: 144575:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4407.672598] Lustre: 144624:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4410.145873] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4411.639956] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4417.612944] Lustre: 144770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4419.133966] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4420.410735] Lustre: 144879:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4428.256472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4428.262677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.267972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4428.274194] Lustre: Skipped 1 previous similar message [ 4433.380910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4433.385117] Lustre: Skipped 2 previous similar messages [ 4434.155630] Lustre: server umount lustre-OST0000 complete [ 4438.503113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.524102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4438.530902] Lustre: Skipped 1 previous similar message [ 4438.549089] Lustre: Skipped 1 previous similar message [ 4443.619475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4443.624480] Lustre: Skipped 1 previous similar message [ 4444.719593] Lustre: server umount lustre-MDT0000 complete [ 4448.422889] LustreError: 141955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444333 with bad export cookie 12522667663326308061 [ 4448.425587] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4448.434058] LustreError: 141955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4448.648470] Lustre: server umount lustre-MDT0001 complete [ 4461.088450] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4463.636917] Key type lgssc unregistered [ 4463.939494] LNet: 145936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4463.951669] LNetError: 145936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4463.967118] LNet: Removed LNI 192.168.201.121@tcp [ 4464.805937] Key type .llcrypt unregistered [ 4464.813498] Key type ._llcrypt unregistered [ 4494.499464] Key type ._llcrypt registered [ 4494.501870] Key type .llcrypt registered [ 4494.675225] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4495.855489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4495.971083] alg: No test for adler32 (adler32-zlib) [ 4497.108607] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4497.300836] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4499.063543] Key type lgssc registered [ 4500.447885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4508.976589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4508.986592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.460375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.106413] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4520.941315] LustreError: 147348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.967055] LustreError: 147348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4523.773948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.041400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4529.097956] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4537.029398] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4544.038840] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4552.478702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4553.043787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.126782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4560.201391] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4569.472214] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4569.950511] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4577.266544] Lustre: *** cfs_fail_loc=159, val=0*** [ 4592.901141] Lustre: lustre-MDT0000: Client 8c884917-5742-4313-a345-f758054fd0f6 (at 192.168.201.21@tcp) reconnecting [ 4598.781042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4598.795172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4598.810210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4600.301916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4600.315073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4600.320448] LustreError: Skipped 6 previous similar messages [ 4600.323289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.031798] Lustre: server umount lustre-OST0000 complete [ 4610.535169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.556890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4610.560921] Lustre: Skipped 2 previous similar messages [ 4615.649098] LustreError: 148081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.731755] Lustre: server umount lustre-MDT0000 complete [ 4619.976579] LustreError: 147329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444505 with bad export cookie 13902614898720577738 [ 4619.982678] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4619.988828] LustreError: 147329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4620.386485] Lustre: server umount lustre-MDT0001 complete [ 4631.977620] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4634.878498] Key type lgssc unregistered [ 4635.142663] LNet: 151013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4635.155691] LNetError: 151013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4635.193887] LNet: Removed LNI 192.168.201.121@tcp [ 4636.032982] Key type .llcrypt unregistered [ 4636.034755] Key type ._llcrypt unregistered [ 4665.181180] Key type ._llcrypt registered [ 4665.184874] Key type .llcrypt registered [ 4665.420826] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4667.524663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4667.605765] alg: No test for adler32 (adler32-zlib) [ 4668.771694] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4668.936744] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4670.584275] Key type lgssc registered [ 4671.323209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4679.859991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4679.887759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.336035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4685.827766] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4691.940487] LustreError: 152426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.955648] LustreError: 152426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4695.886783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.189725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4700.867677] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4710.275270] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4717.780327] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4726.493879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4727.081863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4735.127250] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4741.545070] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4743.354105] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4751.851624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4775.209850] Lustre: lustre-OST0000: evicting 1c2bdc54-efc9-436d-a06b-37850fdbb783 (at 192.168.201.21@tcp) by administrative request [ 4808.161803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4808.167706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4808.178841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4808.214772] Lustre: Skipped 1 previous similar message [ 4809.794253] Lustre: server umount lustre-OST0000 complete [ 4818.412640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4818.434746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4818.437337] Lustre: Skipped 1 previous similar message [ 4820.988960] Lustre: server umount lustre-MDT0000 complete [ 4823.521350] LustreError: 152426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.568109] LustreError: 152426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4825.795576] LustreError: 152406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444710 with bad export cookie 6842114015393135237 [ 4825.807750] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4825.824039] LustreError: 152406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4826.169849] Lustre: server umount lustre-MDT0001 complete [ 4844.505973] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 4848.366601] Key type lgssc unregistered [ 4848.686799] LNet: 156537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4848.692930] LNetError: 156537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4848.704866] LNet: Removed LNI 192.168.201.121@tcp [ 4849.846388] Key type .llcrypt unregistered [ 4849.848074] Key type ._llcrypt unregistered [ 4938.424670] Key type ._llcrypt registered [ 4938.427624] Key type .llcrypt registered [ 4938.561438] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 4939.922887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4939.955087] alg: No test for adler32 (adler32-zlib) [ 4940.962607] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4941.177943] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 4942.863180] Key type lgssc registered [ 4943.821514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4951.552565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4951.581955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.903353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.234306] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4962.274672] LustreError: 159023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.300582] LustreError: 159023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4966.002638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.346112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4971.724915] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 4982.297634] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4988.837186] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4996.321817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4996.817566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4997.903181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5004.110438] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5011.190270] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5013.459789] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5026.785513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5026.786597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5026.789449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5026.804047] Lustre: Skipped 1 previous similar message [ 5030.624612] Lustre: server umount lustre-OST0000 complete [ 5037.028402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.074276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5037.090046] Lustre: Skipped 2 previous similar messages [ 5041.378397] Lustre: server umount lustre-MDT0000 complete [ 5042.147202] LustreError: 159756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5042.171033] LustreError: 159756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5045.692837] LustreError: 159767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444930 with bad export cookie 11315872289662035098 [ 5045.702084] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5045.718413] LustreError: 159767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5046.558628] Lustre: server umount lustre-MDT0001 complete [ 5060.128350] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5063.063280] Key type lgssc unregistered [ 5063.539050] LNet: 162569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5063.555793] LNetError: 162569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5063.582784] LNet: Removed LNI 192.168.201.121@tcp [ 5064.742692] Key type .llcrypt unregistered [ 5064.745343] Key type ._llcrypt unregistered [ 5095.012391] Key type ._llcrypt registered [ 5095.018486] Key type .llcrypt registered [ 5095.143322] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5096.168607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5096.362616] alg: No test for adler32 (adler32-zlib) [ 5097.533383] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5097.782456] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5099.464430] Key type lgssc registered [ 5100.563597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5116.369572] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 10:16:41 (1788445001) [ 5135.456714] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 5148.617583] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5193.081749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5193.185820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.695778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5194.746522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5194.892116] Lustre: lustre-MDT0000: new disk, initializing [ 5195.013084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5195.033056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5200.206675] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5213.880413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5214.133872] Lustre: 166832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5214.248363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5214.254389] Lustre: Skipped 1 previous similar message [ 5214.394374] Lustre: lustre-MDT0001: new disk, initializing [ 5214.616559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5214.719819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5214.740087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5219.984630] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5230.837555] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5237.453915] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5245.329581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5245.657866] Lustre: lustre-OST0000: new disk, initializing [ 5245.661843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5245.667370] Lustre: 168801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5245.820834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5252.223509] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5254.767062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5254.790022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5254.920155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5262.542682] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5275.618616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.620249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.630861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.650288] Lustre: Skipped 1 previous similar message [ 5279.552663] Lustre: server umount lustre-OST0000 complete [ 5285.867535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.883480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.888570] Lustre: Skipped 1 previous similar message [ 5285.901966] Lustre: Skipped 2 previous similar messages [ 5290.975882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5290.988418] Lustre: Skipped 1 previous similar message [ 5295.072981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.079926] Lustre: Skipped 1 previous similar message [ 5298.144709] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5298.637959] Lustre: server umount lustre-MDT0000 complete [ 5303.389981] LustreError: 166825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445188 with bad export cookie 2754487389164166440 [ 5303.394353] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.405404] LustreError: 166825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5303.749536] Lustre: server umount lustre-MDT0001 complete [ 5317.445362] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 5320.659403] Key type lgssc unregistered [ 5320.925556] LNet: 170709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5320.934560] LNetError: 170709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5320.966121] LNet: Removed LNI 192.168.201.121@tcp [ 5322.019896] Key type .llcrypt unregistered [ 5322.021957] Key type ._llcrypt unregistered [ 5350.246630] Key type ._llcrypt registered [ 5350.250727] Key type .llcrypt registered [ 5350.400425] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 5352.431924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5352.511614] alg: No test for adler32 (adler32-zlib) [ 5353.730457] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5354.050341] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 5355.823283] Key type lgssc registered [ 5357.777039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5366.212104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5366.250534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.812200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5372.537025] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5376.993507] LustreError: 172130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.036299] LustreError: 172130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5381.770313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5382.173036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5386.394553] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5395.567156] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5403.500442] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5415.314121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5416.049714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5423.556700] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 5432.338283] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5437.563894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5444.903776] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5444.950331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.968244] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5444.974299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.985711] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5446.312352] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5446.347800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.365373] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5446.369424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.395272] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5447.548297] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5447.562766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5447.578654] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5447.590757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.606157] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5448.574399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.797822] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5449.803072] Lustre: Skipped 1 previous similar message [ 5449.819326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.829265] Lustre: Skipped 1 previous similar message [ 5449.832433] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5449.835371] Lustre: Skipped 1 previous similar message [ 5449.839589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.850317] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5449.858397] Lustre: Skipped 1 previous similar message [ 5450.709932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.744028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.681763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.758958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.671908] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5454.675812] Lustre: Skipped 4 previous similar messages [ 5454.687322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.695671] Lustre: Skipped 4 previous similar messages [ 5454.699280] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5454.703753] Lustre: Skipped 4 previous similar messages [ 5454.707986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.717618] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5454.728383] Lustre: Skipped 4 previous similar messages [ 5455.984593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.889174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.799882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.613572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.549484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.549507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.427035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.230437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.119983] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5463.129217] Lustre: Skipped 8 previous similar messages [ 5463.160118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5463.168341] Lustre: Skipped 8 previous similar messages [ 5463.172050] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5463.176201] Lustre: Skipped 8 previous similar messages [ 5463.179382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.187670] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5463.192792] Lustre: Skipped 8 previous similar messages [ 5464.061467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.280156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.307120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.228883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.100939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.269708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.463461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.471068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.666881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.063320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.290894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.796190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.276681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.422898] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5479.427903] Lustre: Skipped 13 previous similar messages [ 5479.458780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.465668] Lustre: Skipped 13 previous similar messages [ 5479.467512] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5479.469459] Lustre: Skipped 13 previous similar messages [ 5479.471299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.477484] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5479.480757] Lustre: Skipped 13 previous similar messages [ 5480.104488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.595712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.179734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.136220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.002559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.061499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.106075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.903684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.799844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.693266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.797875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.721812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.163172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.443473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.499303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.776559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.743602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.071859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.354557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.815482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.912309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.174795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.286616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.466354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.450347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.998503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.285099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.259258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.714754] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5512.736507] Lustre: Skipped 28 previous similar messages [ 5512.769975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5512.805852] Lustre: Skipped 28 previous similar messages [ 5512.823688] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5512.831974] Lustre: Skipped 28 previous similar messages [ 5512.845294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.870823] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5512.887145] Lustre: Skipped 28 previous similar messages [ 5514.382928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.721237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.753770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.083847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.333496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.212612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.157130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.076971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.390750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.472315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.283477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.721989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.021548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.155606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.120925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.400855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.521992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.677922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.535283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.552384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.569471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.525837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.437077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.204651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.750567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.636105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.517851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.778678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.762455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.761641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.696750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.647739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.676934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.619090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.703970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.742268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.625653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.672041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.925424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.835946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.955304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.949466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.946332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.074689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.101621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.131223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.227307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.556844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.735618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.261323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.943578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.343726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.580038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.582405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.713295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.787565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.638508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.852737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.989435] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5576.993919] Lustre: Skipped 58 previous similar messages [ 5577.004175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.009602] Lustre: Skipped 58 previous similar messages [ 5577.013093] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5577.016626] Lustre: Skipped 58 previous similar messages [ 5577.022961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.037387] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5577.043346] Lustre: Skipped 58 previous similar messages [ 5578.254133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.984935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.099928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.416666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.866491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.798071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.286773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.633514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.630843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.436652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.606809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.525503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.449060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.369767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.349664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.251473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.127802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.929917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.956059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.601326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.569744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.843712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.975754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.986576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.201706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.964780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.874453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.825685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.669588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.537940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.423567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.337425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.211955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.180636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.262313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.236936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.305637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.574416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.518939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.420292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.468048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.581993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.745425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.700558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.831918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.569664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.616861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.428091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.703217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.781553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.679322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.555800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.326483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.215831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.166747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.129608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.242872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.140809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.110508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.062209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.144397] LustreError: 172150:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5638.158146] LustreError: 172150:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5638.164537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.318960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.265106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.175897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.034316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.994697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.806970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.635871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.444836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.414114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.226369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.108981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.993843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.057534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.901709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.917167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.851161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.701634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.568377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.616719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.690855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.720086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.778468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.686324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.552673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.447868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.235784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.143305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.082524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.020928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.880638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.868466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.778877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.959641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.125148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.248789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.194710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.095176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.971112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.909841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.837328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.626657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.493031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.358621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.449409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.012732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.727016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.212097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.577177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.127056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.479324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.884265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.174292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.367666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.527055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.839035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.321807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.293508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.648297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.100196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.823784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.186718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.884931] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5705.893351] Lustre: Skipped 122 previous similar messages [ 5705.926788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.958350] Lustre: Skipped 122 previous similar messages [ 5705.996819] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5706.014672] Lustre: Skipped 122 previous similar messages [ 5706.026445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.058486] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5706.072113] Lustre: Skipped 122 previous similar messages [ 5707.733504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.762184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.550118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.174888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.619545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.810292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.317557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.807304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.884853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.344220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.031934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.305514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.372806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.179586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.989867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.924738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.697280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.768376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.473027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.322044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.291931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.504739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.773716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.701913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.671874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.622833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.584879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.533292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.456311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.263717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.168259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.141682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.005491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.893253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.838458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.880617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.846978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.913394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.861577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.877555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.696366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.605894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.387899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.249137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.035278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.955616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.896661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.890058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.609718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.552096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.442706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.503304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.385770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.313415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.362103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.092851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.994372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.633205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.423475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.236150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.139777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.120909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.043875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.872580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.803625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.023271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.110211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.169683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.299753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.539287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.770905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.107457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.098117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.172985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.156801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.397552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.510953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.490736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.193947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.187494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.117300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.107267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.195588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.094598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.898528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.722244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.631188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.434942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.290615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.218860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.024690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.876245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.830600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.599075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.354555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.094786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.844623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.552625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.634400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.619689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.752848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.954642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.758788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.874446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.630060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.692706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.684921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.926155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.960657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.073237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.161070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.358371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.472629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.267750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.259838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.217474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.245905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.096639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.099526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.246700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.400767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.467115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.746430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.523853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.304282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.281134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.526057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.251402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.513167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.434268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.364112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.339380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.347455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.354875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.522512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.671981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.664696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.630107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.550441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.799802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.911201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.778365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.598193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.678634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.531463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.711991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.863793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.909823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.172572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.258481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.238996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.240096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.204890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.065459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.973350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.936084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.968985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.776883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.907151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.823105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.843163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.724135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.755422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.535772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.520035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.595646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.544028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.444632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.332994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.225179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.996662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.947574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.899305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.972980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.066368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.811462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.664578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.582342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.701686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.907572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.055057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.346256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.460185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.438728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.380620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.394522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.354667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.124879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.909970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.941575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.774616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.848313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.866242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.936225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.808591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.707201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.654424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.552409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.562911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.534204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.576552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.567597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.478487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.287069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.148605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.992341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.913983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.837235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.921086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.015061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.955751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.774034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.494420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.356895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.349797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.241691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.117565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.025078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.814315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.666063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.484919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.460258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.319432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.473773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.638144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.898537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.967039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.098857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.948668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.037671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.970918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.997659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.822537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.734926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.743797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.746807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.794585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.655550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.705648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.551523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.404204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.400986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.326067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.373295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.389709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.503043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.461964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.618394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.718536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.050884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.386428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.614419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.467235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.655968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.840321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.714354] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5962.719621] Lustre: Skipped 255 previous similar messages [ 5962.739868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.748919] Lustre: Skipped 255 previous similar messages [ 5962.753286] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5962.756240] Lustre: Skipped 255 previous similar messages [ 5962.767421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.782039] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5962.784725] Lustre: Skipped 255 previous similar messages [ 5964.162047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.405391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.608815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.887928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.048946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.158513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.290882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.364956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.383251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.627157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.681719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.979604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.983951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.981508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.861807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.788988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.758580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.698979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.729654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.632453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.696871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.577928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.637818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.612139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.630823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.754695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.610048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.471155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.370779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.288692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.172627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.091499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.036789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.109068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.942666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.855440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.713924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.724200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.900835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.839721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.969472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.131846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.480408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.639994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.856355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.092354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.932472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.093311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.016682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.880340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.789994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.656247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.589183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.733319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.538956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.774333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.944280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.188131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.335378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.532105] LustreError: 172150:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6025.550712] LustreError: 172150:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6025.571392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.963919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.962185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.058727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.173667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.189737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.176354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.074884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.006820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.894537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.038851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.981360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.909033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.805237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.571398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.454881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.412426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.540265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.338851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.331768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.228963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.193541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.307395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.389344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.307108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.206431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.032098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.936091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.627508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.455972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.345186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.412485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.289774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.344501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.383914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.313745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.074973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.936519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.784655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.872196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.812443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.731946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.684910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.498882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.385031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.269493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.217638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.177466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.178247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.030602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.894103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.791603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.754260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.870732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.979617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.957215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.846935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.710411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.847792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.834813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.850939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.792326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.002767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.121895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.220642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.191845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.294481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.307489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.459679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.862120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.989907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.279114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.254201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.335716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.284565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.254990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.910635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.062902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.333403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.579792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.889280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.018498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.521354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.466779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.695756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.676585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.627434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.851714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.094329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.439653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.771899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.006380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.310770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.018957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.259925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.872798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.932521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.066782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.223588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.261630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.267395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.213323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.392030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.247147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.459088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.463726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.630778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.007453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.395815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.630834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.478367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.356845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.923667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.547797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.306536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.260400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.482917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.363372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.387804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.407842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.353257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.324134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.247427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.280962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.284821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.343551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.531199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.533516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.666121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.618983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.832645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.999486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.913731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.945282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.835449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.236814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.377379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.528069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.624099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.548755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.910824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.057474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.920654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.050056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.943587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.017131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.323386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.777190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.112227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.414104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.337810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.481650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.595862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.385618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.414366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.265737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.430712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.577727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.644975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.907077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.809483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.744542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.502877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.761588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.828799] LustreError: 172150:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6196.844349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.847135] LustreError: 172150:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6197.999878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.843850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.769353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.845812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.848739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.185434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.254109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.292454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.221539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.266792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.244386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.079101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.022792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.096612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.140150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.885673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.837322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.584513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.516084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.406801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.323410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.228524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.071716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.313264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.513444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.779888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.883328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.068358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.112486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.064783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.857813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.780358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.633726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.555688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.328469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.151012] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.923529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.853588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.817229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.892858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.721281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.850596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.774519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.028992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.230582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.061615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.424499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.479292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.521982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.458981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.488417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.495066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.299680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.245685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.064187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.089805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.928555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.803868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.728239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.709900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.595093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.748133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.959047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.253891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.459611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.625230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.622808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.588672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.342238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.335241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.297790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.233359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.216705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.239671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.105060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.170638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.066757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.969197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.771525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.794503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.719622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.665745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.608347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.450935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.356484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.414165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.527477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.363328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.320192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.216972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.172808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.945252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.911864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.806869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.755403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.616692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.601473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.497227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.763254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.683569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.727229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.762680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.837399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.241397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.160436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.240544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.365446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.563728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.684237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.052256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.419302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.755766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.091910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.065079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.159420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.475582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.723077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.692753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.889241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.255214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.444627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.239287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.858597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.593282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.591743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.569746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.756446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.961504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.991354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.932065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.777640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.694783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.506523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.399330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.089425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.969899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.766678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.555610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.515286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.429849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.144173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.153090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.173480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.232928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.188778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.228052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.081848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.094814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.942371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.901143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.985338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.681460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.561269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.482376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.432672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.411994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.384365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.388453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.201334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.922988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.751800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.671123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.416593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.252854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.132981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.039261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.923512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.996946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.999746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.973811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.686791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.571861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.354535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.332679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.046435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.918369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.726136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.502691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.213781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.154863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.824446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.661927] LustreError: 172150:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6373.673410] LustreError: 172150:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6373.695534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.562946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.462302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.307170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.304439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.034785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.841822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.579595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.787552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.641787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.463904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.148624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.210638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.259592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.054624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.105279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.830329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.590662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.434699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.238197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.051151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.754890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.595078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.348862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.154681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.856572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.626596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.390380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.234653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.250417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.242247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.175513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.160302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.109570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.086455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.071254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.941487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.013557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.721358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.562299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.360022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.210064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.062221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.095273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.963122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.031671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.953287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.992948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.923407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.715183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.505361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.462059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.314308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.227462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.039095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.063378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.923384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.744460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.564207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.552505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.474588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.460607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.461433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.438366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.253789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.094246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.002981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.781619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.625681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.452872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.545720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.586972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.507201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.498343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.527844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.466874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.416210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.233936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.333786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.089375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.079355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.120908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.135058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.289964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.363125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.673673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.654363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.520612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.395945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.302815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.122782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.183553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.392677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.462624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.381370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6815.121119] LustreError: 173954:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6815.144763] LustreError: 172150:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7255.523156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7255.535711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.559565] Lustre: Skipped 500 previous similar messages [ 7255.567269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7255.574962] Lustre: Skipped 1 previous similar message [ 7259.619642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7259.631909] Lustre: Skipped 1 previous similar message [ 7261.483445] Lustre: server umount lustre-OST0000 complete [ 7266.280153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7266.295354] LustreError: Skipped 1 previous similar message [ 7266.307133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7269.859057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7269.876074] Lustre: Skipped 1 previous similar message [ 7272.378660] Lustre: server umount lustre-MDT0000 complete [ 7274.979334] LustreError: 172131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7275.018279] LustreError: 172131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7277.323031] LustreError: 224307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447162 with bad export cookie 13438872297736547394 [ 7277.329412] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7277.343026] LustreError: 224307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7277.804666] Lustre: server umount lustre-MDT0001 complete [ 7289.968309] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7292.743834] Key type lgssc unregistered [ 7293.143680] LNet: 227374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7293.151160] LNetError: 227374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7293.161860] LNet: Removed LNI 192.168.201.121@tcp [ 7294.147363] Key type .llcrypt unregistered [ 7294.149974] Key type ._llcrypt unregistered [ 7316.960631] Key type ._llcrypt registered [ 7316.963743] Key type .llcrypt registered [ 7317.039195] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7318.051910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7318.062700] alg: No test for adler32 (adler32-zlib) [ 7319.075647] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7319.252131] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7320.879290] Key type lgssc registered [ 7321.810813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7328.713980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7328.729330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.195444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7334.664351] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7340.521526] LustreError: 228785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.547230] LustreError: 228785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7343.246649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7343.663063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7348.529372] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7356.148739] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7362.102473] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7368.852233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7369.362247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.471517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7376.644309] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7386.132271] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7391.839267] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7393.913419] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.926803] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7393.937664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7393.945863] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7393.946693] LustreError: 228805:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7399.116191] Lustre: lustre-OST0000: Not available for connect from 192.168.201.21@tcp (stopping) [ 7399.394484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7399.409094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7402.098843] Lustre: server umount lustre-OST0000 complete [ 7414.039408] LustreError: 228805:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7436.118839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7436.444253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7439.539710] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7439.563072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7439.566088] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7439.590869] Lustre: Skipped 1 previous similar message [ 7442.590835] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7450.547382] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7452.342977] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7457.390651] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a2248a6d000.ost_server_uuid 50 [ 7458.949202] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a2248a6d000.ost_server_uuid in FULL state after 0 sec [ 7464.417526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7464.425584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7464.445984] Lustre: Skipped 1 previous similar message [ 7464.462801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7464.466309] Lustre: Skipped 2 previous similar messages [ 7465.457914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7465.467395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7465.472868] Lustre: Skipped 1 previous similar message [ 7466.021254] Lustre: server umount lustre-OST0000 complete [ 7470.560092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7470.565702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7470.583144] Lustre: Skipped 2 previous similar messages [ 7475.680595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7475.685527] Lustre: Skipped 2 previous similar messages [ 7476.629745] Lustre: server umount lustre-MDT0000 complete [ 7479.611044] LustreError: 228765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447364 with bad export cookie 1931154476227499081 [ 7479.627267] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7479.851579] Lustre: server umount lustre-MDT0001 complete [ 7489.820663] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 7492.327628] Key type lgssc unregistered [ 7492.623602] LNet: 233987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7492.632197] LNetError: 233987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7492.650387] LNet: Removed LNI 192.168.201.121@tcp [ 7493.478273] Key type .llcrypt unregistered [ 7493.486447] Key type ._llcrypt unregistered [ 7526.913821] Key type ._llcrypt registered [ 7526.918228] Key type .llcrypt registered [ 7527.110859] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7528.389875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7528.539309] alg: No test for adler32 (adler32-zlib) [ 7529.690915] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7530.004537] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 7531.735300] Key type lgssc registered [ 7532.956082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7541.155762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7541.184536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7542.636325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7546.695935] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7551.971868] LustreError: 235715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7551.980678] LustreError: 235715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7555.747816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.313715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7560.501424] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7563.368131] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7571.153023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7571.741058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7579.822619] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7587.177684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7589.351056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7589.533759] Lustre: lustre-OST0001: new disk, initializing [ 7589.541319] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7589.550727] Lustre: 238022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7589.706381] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7596.457933] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7596.713578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7596.730432] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7596.893331] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7608.173696] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7614.347654] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7627.762359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7627.769324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7627.772882] Lustre: Skipped 2 previous similar messages [ 7627.779886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7627.803777] Lustre: Skipped 3 previous similar messages [ 7628.424758] Lustre: server umount lustre-MDT0000 complete [ 7632.311861] LustreError: 235695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447517 with bad export cookie 3509115442322927571 [ 7632.318928] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7632.327880] LustreError: 235695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7632.842695] Lustre: server umount lustre-MDT0001 complete [ 7647.965710] Lustre: server umount lustre-OST0000 complete [ 7648.738892] Lustre: 234976:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447517/real 1788447517] req@ffff939b1b3ddc00 x1875323036721536/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7648.765387] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7648.777287] Lustre: Skipped 1 previous similar message [ 7652.305514] Lustre: server umount lustre-OST0001 complete [ 7674.279327] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7684.608806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7685.081037] LustreError: 241127:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7685.166374] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.875518] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7690.217986] LustreError: 241128:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7695.339736] LustreError: 241127:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.934324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.403270] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7704.963140] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7708.627995] Lustre: 242270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7715.085463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7722.538559] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7725.866522] LustreError: 242623:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.998552] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 7732.214597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7732.533980] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7732.539243] Lustre: Skipped 1 previous similar message [ 7737.857261] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7739.668660] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7748.597815] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7752.875495] Lustre: 244144:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7767.029158] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7767.052668] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7767.067273] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7768.545616] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7770.204178] Lustre: server umount mylustre-MDT0000 complete [ 7773.332319] LustreError: 241103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447658 with bad export cookie 3509115442322930378 [ 7773.341273] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7773.351510] LustreError: 241103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7773.669691] Lustre: server umount mylustre-MDT0001 complete [ 7788.106699] Lustre: server umount mylustre-OST0000 complete [ 7789.008168] Lustre: 234979:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447658/real 1788447658] req@ffff939b2124c700 x1875323036781952/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7789.044511] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.070111] Lustre: Skipped 3 previous similar messages [ 7792.316303] Lustre: server umount mylustre-OST0001 complete [ 7814.799201] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7824.102808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7824.400618] LustreError: 246459:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.413693] LustreError: 246459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7824.454813] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.752591] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7836.805260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7841.492531] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7844.629911] Lustre: 247601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7851.217562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7856.915461] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7862.897289] LustreError: 247956:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7862.918307] LustreError: 247956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7864.243451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7869.554394] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 7869.574152] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49792) [ 7870.306572] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7877.621513] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7881.139101] Lustre: 249472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7888.354565] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7888.359408] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.381149] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7888.385709] Lustre: Skipped 3 previous similar messages [ 7894.270769] Lustre: server umount tfs-MDT0000 complete [ 7895.533847] LustreError: 246459:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7895.560648] LustreError: 246459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7898.203906] LustreError: 246437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447783 with bad export cookie 3509115442322932268 [ 7898.204458] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7898.219056] LustreError: 246437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7898.732630] Lustre: server umount tfs-MDT0001 complete [ 7912.886841] Lustre: server umount tfs-OST0000 complete [ 7915.999136] Lustre: 234976:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447785/real 1788447785] req@ffff939b08a8ed80 x1875323036830208/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7916.045673] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.056231] Lustre: Skipped 3 previous similar messages [ 7917.059553] Lustre: server umount tfs-OST0001 complete [ 7943.376188] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 7954.604908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7955.154556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7955.159503] Lustre: Skipped 3 previous similar messages [ 7960.263358] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7960.567515] LustreError: 251724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7960.590080] LustreError: 251724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7968.634594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.277882] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7976.438527] Lustre: 252866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7983.812713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7990.504912] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 7998.561052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8000.881142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49824) [ 8000.906617] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 8004.765807] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8012.394960] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8016.220214] Lustre: 254737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8024.400368] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 11:05:08 (1788447908) [ 8041.956042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8041.961816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8041.974037] Lustre: Skipped 3 previous similar messages [ 8041.994614] Lustre: Skipped 7 previous similar messages [ 8048.237155] Lustre: server umount lustre-MDT0000 complete [ 8054.365373] LustreError: 251701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447939 with bad export cookie 3509115442322933934 [ 8054.368642] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8054.386965] LustreError: 251701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8057.337262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8057.349543] Lustre: Skipped 5 previous similar messages [ 8060.819783] Lustre: server umount lustre-MDT0001 complete [ 8079.327123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8079.465163] Lustre: server umount lustre-OST0000 complete [ 8098.271426] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8098.437959] Lustre: server umount lustre-OST0001 complete [ 8104.622325] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 8114.259295] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8151.440585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8151.672285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8151.703647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8151.784650] Lustre: lustre-MDT0000: new disk, initializing [ 8151.872240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8151.879298] Lustre: Skipped 3 previous similar messages [ 8151.891569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8157.004049] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8168.344507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8168.571533] Lustre: 257925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8168.657918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8168.662720] Lustre: Skipped 1 previous similar message [ 8168.752964] Lustre: lustre-MDT0001: new disk, initializing [ 8168.868810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8168.881554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8173.868726] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8184.342365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8184.712269] Lustre: lustre-OST0000: new disk, initializing [ 8184.716518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8184.723723] Lustre: 259507:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8186.438981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8186.442914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8186.623664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8192.425176] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8203.893567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8204.199989] Lustre: lustre-OST0001: new disk, initializing [ 8204.209782] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8204.226879] Lustre: 260379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8205.688875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8205.698602] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8205.824991] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8210.765745] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8223.484033] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 11:08:28 (1788448108) [ 8237.785740] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 11:08:42 (1788448122) [ 8253.408334] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8253.409316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.416895] LustreError: Skipped 1 previous similar message [ 8253.417826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8253.417831] Lustre: Skipped 2 previous similar messages [ 8253.458819] Lustre: Skipped 3 previous similar messages [ 8258.567652] Lustre: server umount lustre-OST0000 complete [ 8262.114022] LustreError: 259598:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8262.132576] LustreError: 259598:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8266.209566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8268.630125] Lustre: server umount lustre-MDT0000 complete [ 8273.108966] LustreError: 257917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448158 with bad export cookie 3509115442322935488 [ 8273.158167] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8273.197308] LustreError: 257917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8279.583468] Lustre: server umount lustre-MDT0001 complete [ 8292.041357] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8366.493502] Lustre: server umount lustre-OST0001 complete [ 8373.910691] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 8392.258144] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8428.236348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.435365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8428.467535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8428.563447] Lustre: lustre-MDT0000: new disk, initializing [ 8428.646362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8428.668172] Lustre: Skipped 3 previous similar messages [ 8428.685652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8433.032515] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8443.672418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8443.803418] Lustre: 265233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8444.118019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8448.251760] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8457.990593] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8463.790859] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8470.931849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8471.302535] Lustre: lustre-OST0000: new disk, initializing [ 8471.307692] Lustre: Skipped 1 previous similar message [ 8471.311782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8471.324789] Lustre: Skipped 2 previous similar messages [ 8471.336680] Lustre: 267200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8472.540085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8472.569970] Lustre: Skipped 1 previous similar message [ 8472.599093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8472.733161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8478.043508] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8490.808564] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8503.269633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8503.283505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.296129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8503.296138] Lustre: Skipped 9 previous similar messages [ 8503.351587] Lustre: Skipped 5 previous similar messages [ 8507.475897] Lustre: server umount lustre-OST0000 complete [ 8517.967688] Lustre: server umount lustre-MDT0000 complete [ 8518.633599] LustreError: 267244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8518.657349] LustreError: 267244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 8522.484386] LustreError: 265225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448407 with bad export cookie 3509115442322936958 [ 8522.495230] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.506825] LustreError: 265225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8523.035579] Lustre: server umount lustre-MDT0001 complete [ 8538.005549] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8540.799767] Key type lgssc unregistered [ 8541.073646] LNet: 268963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8541.079886] LNetError: 268963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8541.094921] LNet: Removed LNI 192.168.201.121@tcp [ 8541.949416] Key type .llcrypt unregistered [ 8541.951157] Key type ._llcrypt unregistered [ 8569.864384] Key type ._llcrypt registered [ 8569.867310] Key type .llcrypt registered [ 8569.994247] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8571.097748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8571.337106] alg: No test for adler32 (adler32-zlib) [ 8572.435784] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8572.737273] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8574.463205] Key type lgssc registered [ 8575.700831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8583.376296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8583.392290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.875889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.950633] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8594.405325] LustreError: 270423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8594.420864] LustreError: 270423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8598.030565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8598.397868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.931972] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8611.001855] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8617.825145] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8624.846811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8625.290334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8629.441061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8631.623391] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8639.013527] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8640.028363] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8643.117588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8643.134137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8648.169941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8648.176659] Lustre: Skipped 1 previous similar message [ 8653.289798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8653.299752] Lustre: Skipped 1 previous similar message [ 8656.351265] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8656.512632] Lustre: server umount lustre-OST0000 complete [ 8663.528199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8663.547656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8663.555343] Lustre: Skipped 2 previous similar messages [ 8666.500100] Lustre: server umount lustre-MDT0000 complete [ 8668.643722] LustreError: 270424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8668.663734] LustreError: 270424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8669.902197] LustreError: 270403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448554 with bad export cookie 10454485801265511647 [ 8669.912174] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8669.917290] LustreError: 270403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8670.525292] Lustre: server umount lustre-MDT0001 complete [ 8679.637167] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8682.119372] Key type lgssc unregistered [ 8682.368709] LNet: 274004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8682.371449] LNetError: 274004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8683.440737] LNet: Removed LNI 192.168.201.121@tcp [ 8684.033380] Key type .llcrypt unregistered [ 8684.035249] Key type ._llcrypt unregistered [ 8701.773741] Key type ._llcrypt registered [ 8701.781908] Key type .llcrypt registered [ 8701.885589] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8702.584684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8702.655631] alg: No test for adler32 (adler32-zlib) [ 8703.709561] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8703.911576] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8705.591586] Key type lgssc registered [ 8706.709096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8715.854290] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8715.880059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8731.423904] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8736.543452] LustreError: 275194:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.121@tcp: failed processing log, type 4: rc = -110 [ 8763.231476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.417225] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 8777.560172] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8779.789414] Lustre: server umount lustre-OST0000 complete [ 8798.303349] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 8801.067875] Key type lgssc unregistered [ 8801.305980] LNet: 276824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8801.312745] LNetError: 276824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8801.334897] LNet: Removed LNI 192.168.201.121@tcp [ 8802.201186] Key type .llcrypt unregistered [ 8802.202923] Key type ._llcrypt unregistered [ 8841.073321] Key type ._llcrypt registered [ 8841.078358] Key type .llcrypt registered [ 8841.188535] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 8842.093548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8842.200717] alg: No test for adler32 (adler32-zlib) [ 8843.240661] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8843.518758] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 8845.176162] Key type lgssc registered [ 8846.147745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8858.877560] loop0: detected capacity change from 0 to 204800000 [ 8859.069150] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8859.076574] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8859.356718] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8860.323728] loop0: detected capacity change from 0 to 204800000 [ 8860.491801] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8860.501674] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8860.831136] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8861.819960] loop0: detected capacity change from 0 to 204800000 [ 8861.955534] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8861.972032] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8862.095708] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8863.052658] loop0: detected capacity change from 0 to 204800000 [ 8863.266859] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8864.296776] loop0: detected capacity change from 0 to 204800000 [ 8864.326799] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8868.201313] loop0: detected capacity change from 0 to 204800000 [ 8868.246141] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8871.803967] loop0: detected capacity change from 0 to 204800000 [ 8871.840563] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8875.384543] loop0: detected capacity change from 0 to 204800000 [ 8875.407817] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8879.405308] loop0: detected capacity change from 0 to 204800000 [ 8879.560728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8879.609352] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8879.666591] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8879.673486] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8879.684103] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8879.690673] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8879.697319] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8879.703521] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8879.709402] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8879.716226] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8879.721236] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8879.726422] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8879.733305] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8879.740804] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8879.748643] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8879.756118] Lustre: 280233:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8885.567771] Lustre: server umount MGS complete [ 8886.559619] loop0: detected capacity change from 0 to 204800000 [ 8886.645217] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.147291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8887.211800] Lustre: lustre-MDT0000: Aborting client recovery [ 8888.349809] loop1: detected capacity change from 0 to 204800000 [ 8888.468553] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.551993] Lustre: 280667:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8888.564529] Lustre: 280667:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8888.577406] Lustre: 280667:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8888.591463] Lustre: 280667:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8888.602856] Lustre: 280667:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8888.616301] Lustre: 280667:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8888.624669] Lustre: 280667:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8888.652324] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8888.815944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8888.859545] Lustre: lustre-MDT0001: Aborting client recovery [ 8889.642788] loop2: detected capacity change from 0 to 204800000 [ 8889.780046] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8889.833497] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8889.845768] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8889.853451] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8889.868744] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8889.876890] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8889.883808] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8889.892888] Lustre: 280746:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8889.985250] Lustre: Found index 0 for lustre-OST0000, updating log [ 8890.012066] Lustre: 280765:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8890.075148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8890.090740] Lustre: lustre-OST0000: Aborting recovery [ 8890.959375] loop3: detected capacity change from 0 to 204800000 [ 8891.102783] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8891.242929] Lustre: 280830:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8891.270194] Lustre: 280830:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8891.289213] Lustre: 280830:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8891.300246] Lustre: 280830:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8891.317229] Lustre: 280830:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8891.326235] Lustre: 280830:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8891.338954] Lustre: 280830:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8891.354363] Lustre: Found index 1 for lustre-OST0001, updating log [ 8891.378266] Lustre: 280837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8891.450770] Lustre: lustre-OST0001: Aborting recovery [ 8896.516972] LustreError: 280611:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -108 [ 8897.517975] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8897.519620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8897.569060] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8897.573219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8898.552519] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8898.556508] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8899.951355] LustreError: 280578:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 673us: rc = -2 [ 8900.471713] LustreError: 280578:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 381us: rc = -2 [ 8900.485768] LustreError: 280578:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 8913.889852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.890729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8913.913538] Lustre: Skipped 3 previous similar messages [ 8913.924540] Lustre: Skipped 3 previous similar messages [ 8919.008688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8919.023124] Lustre: Skipped 3 previous similar messages [ 8919.620332] Lustre: server umount lustre-MDT0000 complete [ 8920.539158] LustreError: 280562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448805 with bad export cookie 2085361757349577950 [ 8920.552503] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8920.554219] LustreError: 280562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8924.128898] LustreError: 280575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8924.129907] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8924.131938] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8924.159068] LustreError: 280575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8924.201608] Lustre: Skipped 1 previous similar message [ 8925.152089] LustreError: 281073:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8925.167254] LustreError: 281073:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8927.049375] Lustre: server umount lustre-MDT0001 complete [ 8934.207971] Lustre: server umount lustre-OST0000 complete [ 8941.660567] Lustre: server umount lustre-OST0001 complete [ 8950.561321] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 11:20:35 (1788448835) [ 8980.085141] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 8991.122439] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9028.997221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.207304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9029.238739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9029.324701] Lustre: lustre-MDT0000: new disk, initializing [ 9029.401482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9029.404324] Lustre: Skipped 1 previous similar message [ 9029.423633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9033.667980] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9046.035050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9046.175477] Lustre: 284903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9046.229458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9046.233143] Lustre: Skipped 1 previous similar message [ 9046.294747] Lustre: lustre-MDT0001: new disk, initializing [ 9046.384493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9046.451223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9046.471091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9051.882390] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9061.868191] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9067.699224] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9073.915564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9074.232266] Lustre: lustre-OST0000: new disk, initializing [ 9074.238629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9074.254564] Lustre: 286875:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9074.345493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9080.138358] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9083.973224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9083.994077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9084.152884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9089.614582] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9094.548662] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9104.227602] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9111.996226] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9119.772127] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9171.434590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9171.440078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9171.454723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9171.455323] LustreError: Skipped 1 previous similar message [ 9171.465942] Lustre: Skipped 4 previous similar messages [ 9174.225383] Lustre: server umount lustre-OST0000 complete [ 9179.622302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9179.629778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9179.635345] Lustre: Skipped 1 previous similar message [ 9179.639717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9184.642933] Lustre: server umount lustre-MDT0000 complete [ 9186.785647] LustreError: 284916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9186.799277] LustreError: 284916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9187.949613] LustreError: 284897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449073 with bad export cookie 2085361757349580701 [ 9187.960767] LustreError: 284897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9187.961858] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9188.405980] Lustre: server umount lustre-MDT0001 complete [ 9196.387644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9200.602146] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9206.984754] Lustre: server umount MGS complete [ 9220.814039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9221.348321] LustreError: 290769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9221.382395] LustreError: 290782:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9221.492488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.482985] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9226.720762] LustreError: 290770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.335903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9239.692551] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9247.439441] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9252.905094] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9259.227852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9259.616058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9259.629683] Lustre: Skipped 1 previous similar message [ 9266.271853] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9275.504751] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9307.107512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9307.122696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9307.136837] Lustre: Skipped 1 previous similar message [ 9307.147136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9307.153563] Lustre: Skipped 2 previous similar messages [ 9310.611366] Lustre: server umount lustre-OST0000 complete [ 9317.866749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9317.877054] LustreError: Skipped 1 previous similar message [ 9317.886164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9317.900548] Lustre: Skipped 1 previous similar message [ 9320.601518] Lustre: server umount lustre-MDT0000 complete [ 9323.490935] LustreError: 290765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9323.514385] LustreError: 290765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9323.980327] LustreError: 290749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449209 with bad export cookie 2085361757349582654 [ 9323.984135] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9323.994770] LustreError: 290749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9324.327281] Lustre: server umount lustre-MDT0001 complete [ 9335.696525] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 9339.667450] Key type lgssc unregistered [ 9339.947565] LNet: 294711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9339.962135] LNetError: 294711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9339.997038] LNet: Removed LNI 192.168.201.121@tcp [ 9341.439281] Key type .llcrypt unregistered [ 9341.441887] Key type ._llcrypt unregistered [ 9386.583449] Key type ._llcrypt registered [ 9386.585468] Key type .llcrypt registered [ 9386.807352] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 9405.578331] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9407.104491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9407.213933] alg: No test for adler32 (adler32-zlib) [ 9408.385464] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9408.619809] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 9410.279273] Key type lgssc registered [ 9411.397803] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9443.635927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9443.653044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9444.881309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9444.913644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9445.003351] Lustre: lustre-MDT0000: new disk, initializing [ 9445.092764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9445.101560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9449.738145] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9461.124717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.299428] Lustre: 298669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9461.408196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9461.415525] Lustre: Skipped 1 previous similar message [ 9461.606617] Lustre: lustre-MDT0001: new disk, initializing [ 9461.857073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9461.925080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9461.943104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9466.992092] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9478.320761] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9484.335747] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9491.370229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9491.667296] Lustre: lustre-OST0000: new disk, initializing [ 9491.670199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9491.673687] Lustre: 300636:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9491.738238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9497.730504] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9499.201564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9499.209857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9499.344496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9507.935944] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9516.059762] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9527.875754] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9534.467526] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9540.228239] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9581.540683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9581.541810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.556746] LustreError: Skipped 1 previous similar message [ 9581.559287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9581.590121] Lustre: Skipped 1 previous similar message [ 9583.903695] Lustre: server umount lustre-OST0000 complete [ 9589.730571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9589.734476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9589.755090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.760517] Lustre: Skipped 1 previous similar message [ 9591.779449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9591.779882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9593.667503] Lustre: server umount lustre-MDT0000 complete [ 9597.049133] LustreError: 298660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449482 with bad export cookie 12278374490553689514 [ 9597.050841] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9597.062145] LustreError: 298660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9597.428345] Lustre: server umount lustre-MDT0001 complete [ 9606.319187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9610.937483] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9616.054900] Lustre: server umount MGS complete [ 9626.846696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9627.392600] LustreError: 304503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9627.416029] LustreError: 304503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9627.423287] LustreError: 304516:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9627.490251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9631.520459] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9632.736357] LustreError: 304504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9637.865905] LustreError: 304503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9640.977264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.592953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9647.743244] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9657.806781] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9664.104677] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9670.495795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9670.902153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9677.052186] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9685.852348] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9717.221273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9717.226852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9717.242513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9717.247806] Lustre: Skipped 2 previous similar messages [ 9722.336681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9722.346294] Lustre: Skipped 1 previous similar message [ 9723.102955] Lustre: server umount lustre-OST0000 complete [ 9727.458087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9727.472134] Lustre: Skipped 1 previous similar message [ 9732.579327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9732.587205] Lustre: Skipped 3 previous similar messages [ 9741.279377] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9741.468431] Lustre: server umount lustre-MDT0000 complete [ 9742.822828] LustreError: 304503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9742.838245] LustreError: 304503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9744.717416] LustreError: 307229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449629 with bad export cookie 12278374490553691327 [ 9744.725243] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9745.102145] Lustre: server umount lustre-MDT0001 complete [ 9755.198414] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [ 9757.739684] Key type lgssc unregistered [ 9758.098284] LNet: 308450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9758.112189] LNetError: 308450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9758.128061] LNet: Removed LNI 192.168.201.121@tcp [ 9759.040102] Key type .llcrypt unregistered [ 9759.046977] Key type ._llcrypt unregistered [ 9778.961640] Key type ._llcrypt registered [ 9778.962948] Key type .llcrypt registered [ 9779.032892] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [ 9793.079705] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9794.427752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9794.443749] alg: No test for adler32 (adler32-zlib) [ 9795.566260] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9795.849266] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [ 9797.543231] Key type lgssc registered [ 9798.609681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9829.204560] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [ 9847.692168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9847.709174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9848.992579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9849.053932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9849.223748] Lustre: lustre-MDT0000: new disk, initializing [ 9849.411437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9849.461036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9855.502704] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9868.479331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9868.691101] Lustre: lustre-OST0000: new disk, initializing [ 9868.694511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9868.697429] Lustre: Skipped 1 previous similar message [ 9868.702138] Lustre: 312499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9868.788114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9877.207343] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [ 9878.703704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9878.713681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9878.754103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10728.929384] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10728.941376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10732.679199] Lustre: server umount lustre-MDT0000 complete [10736.694210] LustreError: 311683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450621 with bad export cookie 14934297561504332509 [10736.695128] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10736.702195] LustreError: 311683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10736.937240] Lustre: server umount lustre-OST0000 complete [10764.502867] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [10775.730243] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [10815.349556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10815.727730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10815.752792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10815.913938] Lustre: lustre-MDT0000: new disk, initializing [10816.098340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10816.137782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10820.980343] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10831.844842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10832.051349] Lustre: 316832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10832.124198] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10832.132070] Lustre: Skipped 1 previous similar message [10832.211766] Lustre: lustre-MDT0001: new disk, initializing [10832.305477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10832.350867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10832.363982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10836.377937] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10847.045520] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10853.118741] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10860.259873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10860.476754] Lustre: lustre-OST0000: new disk, initializing [10860.480867] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10860.492604] Lustre: 318803:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10860.646966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10867.462608] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10868.395416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10868.410474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10868.505282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10876.323582] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10880.188787] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10880.280402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10883.934483] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10884.197640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10889.189246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.191201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10889.195193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10892.807764] Lustre: server umount lustre-OST0000 complete [10898.921055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10898.934148] LustreError: Skipped 1 previous similar message [10898.938400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10898.952424] Lustre: Skipped 1 previous similar message [10898.967452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10898.972967] Lustre: Skipped 1 previous similar message [10902.738838] Lustre: server umount lustre-MDT0000 complete [10906.188146] LustreError: 316826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450791 with bad export cookie 14934297561507273328 [10906.200066] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10906.200136] LustreError: 316826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10906.480679] Lustre: server umount lustre-MDT0001 complete [10911.145605] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10913.552369] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 11:53:17 (1788450797) [10922.012119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10922.632299] LustreError: 321117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10922.655660] LustreError: 321117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10922.749563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10927.253737] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10928.097092] LustreError: 321118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.200298] LustreError: 321117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.849273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10936.165891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10940.655146] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10948.143197] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10954.340615] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10961.036152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10961.313606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10966.589323] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10973.087258] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10979.225845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10979.390805] Lustre: lustre-OST0001: new disk, initializing [10979.394395] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10979.402075] Lustre: 323953:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10979.491835] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10986.175588] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [10988.126935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10988.134893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10988.221442] LustreError: 324409:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10988.236138] LustreError: 324409:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10989.281123] LustreError: 324417:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10989.303913] LustreError: 324417:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10990.305794] LustreError: 324409:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10992.358690] LustreError: 324409:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10992.375298] LustreError: 324409:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10996.518983] LustreError: 324409:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10996.536883] LustreError: 324409:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10996.641678] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10996.858379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11003.729752] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a225046d000.ost_server_uuid 50 [11004.833552] LustreError: 324409:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11004.851787] LustreError: 324409:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [11005.713778] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a225046d000.ost_server_uuid in FULL state after 0 sec [11012.563311] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a225046d000.ost_server_uuid 50 [11014.180755] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a225046d000.ost_server_uuid in IDLE state after 0 sec [11016.204679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11029.492240] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11029.498371] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11029.519162] Lustre: Skipped 2 previous similar messages [11029.544324] Lustre: Skipped 3 previous similar messages [11034.600332] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11034.607373] Lustre: Skipped 2 previous similar messages [11042.997121] Lustre: lustre-OST0001: Not available for connect from 192.168.201.21@tcp (stopping) [11043.007336] Lustre: Skipped 3 previous similar messages [11043.295813] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11043.398598] Lustre: server umount lustre-OST0001 complete [11044.838290] LustreError: 322938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11044.860599] LustreError: 322938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11049.451076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11049.456851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11049.461958] LustreError: Skipped 1 previous similar message [11049.485304] Lustre: Skipped 1 previous similar message [11049.972781] LustreError: 323565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.985119] LustreError: 323565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11055.044143] Lustre: server umount lustre-OST0000 complete [11059.168830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11059.175447] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.187390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11059.193252] Lustre: Skipped 2 previous similar messages [11065.188721] Lustre: server umount lustre-MDT0000 complete [11065.313723] LustreError: 321117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11065.333591] LustreError: 321117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11069.107190] LustreError: 321099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450954 with bad export cookie 14934297561507274210 [11069.116424] LustreError: 321099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11069.131853] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11069.460362] Lustre: server umount lustre-MDT0001 complete [11079.713669] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [11082.455722] Key type lgssc unregistered [11082.787388] LNet: 326644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11082.793234] LNetError: 326644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11082.824262] LNet: Removed LNI 192.168.201.121@tcp [11083.639740] Key type .llcrypt unregistered [11083.641627] Key type ._llcrypt unregistered [11107.544817] Key type ._llcrypt registered [11107.546134] Key type .llcrypt registered [11107.693982] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11108.528755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11108.663172] alg: No test for adler32 (adler32-zlib) [11109.751598] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11109.944262] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [11111.591213] Key type lgssc registered [11112.466841] Lustre: Echo OBD driver; http://www.lustre.org/ [11118.883492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11118.909969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11120.421507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11124.253887] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11130.737954] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11130.855811] LustreError: 328060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.865749] LustreError: 328060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11135.975902] LustreError: 328061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11137.162900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11137.405239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11141.073199] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11150.077425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11150.323437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11155.835892] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11162.614673] LustreError: 329743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.441090] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11165.755837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11165.755903] LustreError: 330513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11169.625856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11169.780547] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11174.906625] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11175.781399] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11184.011667] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11190.921387] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11191.099214] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11297.767392] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11297.773387] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11297.784323] LustreError: Skipped 1 previous similar message [11297.786413] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11297.803372] Lustre: Skipped 1 previous similar message [11302.417357] Lustre: server umount lustre-OST0001 complete [11302.880783] LustreError: 329741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.905937] LustreError: 329741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11309.026556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11309.033436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11309.050345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11309.061117] Lustre: Skipped 1 previous similar message [11311.074244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11311.084786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11311.106313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11313.123335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11313.123651] LustreError: 330513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11313.131250] Lustre: Skipped 1 previous similar message [11313.145832] LustreError: 330513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11314.650385] Lustre: server umount lustre-OST0000 complete [11321.826448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11321.830130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11321.836867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11333.087120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11333.223177] Lustre: server umount lustre-MDT0000 complete [11333.600024] LustreError: 328061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11333.620645] LustreError: 328061:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11336.983966] LustreError: 331742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451222 with bad export cookie 10203929532353109467 [11336.989910] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11336.992768] LustreError: 331742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11337.367893] Lustre: server umount lustre-MDT0001 complete [11347.124614] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [11349.705384] Key type lgssc unregistered [11349.994552] LNet: 333003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11350.013214] LNetError: 333003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11350.048547] LNet: Removed LNI 192.168.201.121@tcp [11350.977699] Key type .llcrypt unregistered [11350.980671] Key type ._llcrypt unregistered [11400.047764] Key type ._llcrypt registered [11400.052500] Key type .llcrypt registered [11400.196411] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11401.742128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11401.772492] alg: No test for adler32 (adler32-zlib) [11402.821096] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11403.166978] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [11404.903171] Key type lgssc registered [11406.141771] Lustre: Echo OBD driver; http://www.lustre.org/ [11415.265882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11415.298927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.711255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11420.999300] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11426.277763] LustreError: 335303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11426.286365] LustreError: 335303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11429.803637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.291225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11434.307518] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11437.224171] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11444.335965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11444.709318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11451.356641] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11455.973914] LustreError: 336792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11460.168589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11460.471995] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11461.517666] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11461.699604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11468.291674] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11478.010598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11483.841462] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11496.935274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11496.939463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11496.951866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11497.450578] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11497.459814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11497.470390] Lustre: Skipped 2 previous similar messages [11497.487468] Lustre: Skipped 3 previous similar messages [11500.999693] Lustre: server umount lustre-MDT0000 complete [11502.562933] LustreError: 335299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.592671] LustreError: 335299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11504.827454] LustreError: 335286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451389 with bad export cookie 10673526385302196921 [11504.831475] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11504.846649] LustreError: 335286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11505.233097] Lustre: server umount lustre-MDT0001 complete [11519.363838] Lustre: server umount lustre-OST0000 complete [11533.108876] Lustre: server umount lustre-OST0001 complete [11549.200985] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11558.158437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11558.426613] LustreError: 340567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11558.489601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11562.381244] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11563.489421] LustreError: 340568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11570.475369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11570.887376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11574.591076] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11577.232367] Lustre: 341707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11583.231781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11583.468161] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [11583.482218] LustreError: 342063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.503598] LustreError: 342063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11589.120861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11589.270304] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11597.209850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11597.472808] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11597.486730] Lustre: Skipped 1 previous similar message [11602.938080] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11604.246528] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11612.729276] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11616.985400] Lustre: 343578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11621.869746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11621.875863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11621.888617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11628.021260] Lustre: server umount lustre-MDT0000 complete [11628.542472] LustreError: 343441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.565486] LustreError: 343441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11631.781301] LustreError: 340549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451516 with bad export cookie 10673526385302198622 [11631.801563] LustreError: 340549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11631.810921] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11632.172493] Lustre: server umount lustre-MDT0001 complete [11645.300435] Lustre: server umount lustre-OST0000 complete [11649.759242] Lustre: 334566:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451518/real 1788451518] req@ffff939b0250b100 x1875327097878400/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11649.795616] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.806749] Lustre: Skipped 3 previous similar messages [11650.233829] Lustre: server umount lustre-OST0001 complete [11653.895633] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11656.352171] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11659.229781] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 12:05:43 (1788451543) [11686.812331] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11696.792706] loop0: detected capacity change from 0 to 1099511627776 [11697.532780] SGI XFS with ACLs, security attributes, quota, no debug enabled [11697.541517] *** VALIDATE xfs *** [11697.555889] XFS (loop0): Mounting V5 Filesystem [11697.597617] XFS (loop0): Ending clean mount [11699.092973] loop1: detected capacity change from 0 to 18691697672192 [11806.433562] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [11818.616616] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [11858.409370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11858.621116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11858.647503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11858.705306] Lustre: lustre-MDT0000: new disk, initializing [11858.751422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11858.766529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11863.230709] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11874.089786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11874.312846] Lustre: 348716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11874.411825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11874.415502] Lustre: Skipped 1 previous similar message [11874.510703] Lustre: lustre-MDT0001: new disk, initializing [11874.702689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11874.717570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11880.642978] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11890.758631] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11896.592485] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11903.678294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11903.914598] Lustre: lustre-OST0000: new disk, initializing [11903.917473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11903.922606] Lustre: 350683:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11905.856745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11905.875170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11906.005895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11909.887095] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11919.246838] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11923.796211] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11924.058236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11929.214509] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11929.418622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11931.623844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11931.624276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11931.624956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11931.624961] Lustre: Skipped 4 previous similar messages [11931.634935] Lustre: Skipped 1 previous similar message [11931.651762] LustreError: Skipped 1 previous similar message [11937.852724] Lustre: server umount lustre-OST0000 complete [11941.867214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11941.877541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11941.880235] Lustre: Skipped 3 previous similar messages [11946.464492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11948.156705] Lustre: server umount lustre-MDT0000 complete [11952.098661] LustreError: 349563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11952.114534] LustreError: 349563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11952.192575] LustreError: 348707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451837 with bad export cookie 10673526385302200176 [11952.203283] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11952.600836] Lustre: server umount lustre-MDT0001 complete [11961.777587] XFS (loop0): Unmounting Filesystem [11966.301861] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 12:10:51 (1788451851) [11973.895570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11974.654472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11974.660370] Lustre: Skipped 2 previous similar messages [11979.698460] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [11988.542734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11994.049142] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12002.783709] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12008.847692] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12016.945570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12024.203990] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12031.294116] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12044.257274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12044.262693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12044.266386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12044.275938] Lustre: Skipped 2 previous similar messages [12044.277728] Lustre: Skipped 2 previous similar messages [12049.977806] Lustre: server umount lustre-OST0000 complete [12059.558403] Lustre: server umount lustre-MDT0000 complete [12059.626039] LustreError: 354558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.635873] LustreError: 354558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12063.900946] LustreError: 353225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451948 with bad export cookie 10673526385302201058 [12063.908346] LustreError: 353225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12063.915728] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12064.128323] Lustre: server umount lustre-MDT0001 complete [12075.174122] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [12077.884358] Key type lgssc unregistered [12078.204955] LNet: 356870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12078.217954] LNetError: 356870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12079.269822] LNet: Removed LNI 192.168.201.121@tcp [12080.119190] Key type .llcrypt unregistered [12080.121108] Key type ._llcrypt unregistered [12109.714158] Key type ._llcrypt registered [12109.722704] Key type .llcrypt registered [12109.831996] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [12111.009019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12111.248815] alg: No test for adler32 (adler32-zlib) [12112.459469] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12112.825899] LNet: Added LNI 192.168.201.121@tcp [8/256/0/180] [12114.503601] Key type lgssc registered [12115.682863] Lustre: Echo OBD driver; http://www.lustre.org/ [12123.547973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12123.582379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12125.039685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12130.428736] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12135.403570] LustreError: 358437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.425825] LustreError: 358437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12140.513709] LustreError: 358438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.578118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12140.982454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12145.792458] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12153.634934] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12159.632333] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12168.472210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12168.948432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12175.452438] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12183.398315] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12192.225537] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12192.236986] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12192.266950] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12193.951370] Lustre: server umount lustre-MDT0001 complete [12195.809318] LustreError: 358437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.820273] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12200.928947] LustreError: 358437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.957628] LustreError: 358437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12205.775980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12205.916599] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12205.933801] Lustre: Found index 1 for lustre-MDT0001, updating log [12205.979442] Lustre: 358427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12206.048462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12206.056754] Lustre: Skipped 1 previous similar message [12206.230262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12211.058908] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12211.686406] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12212.235520] LustreError: 361943:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12212.260329] LustreError: 361943:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12212.275191] LustreError: 358429:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12212.404939] LustreError: 361952:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -17 [12212.426648] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12212.426648] [12221.931069] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12221.932476] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12221.944920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12221.961559] Lustre: Skipped 1 previous similar message [12227.053311] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12227.055867] Lustre: Skipped 1 previous similar message [12232.161375] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12232.164268] Lustre: Skipped 2 previous similar messages [12235.231201] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12235.427320] Lustre: server umount lustre-MDT0001 complete [12237.281951] LustreError: 358438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12237.292970] LustreError: 358438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12246.920212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12247.030367] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12247.037755] Lustre: Found index 1 for lustre-MDT0001, updating log [12247.086487] Lustre: 358426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12247.315489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12251.978706] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12252.650897] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12255.240923] LustreError: 363095:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12255.249226] LustreError: 363095:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12255.266832] LustreError: 358429:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12255.269949] LustreError: 358429:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12255.373827] LustreError: 363115:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12255.382234] LustreError: 363115:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12255.385298] LustreError: 363115:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -17 [12255.394346] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.121@tcp [12255.394346] [12557.797426] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12557.818926] Lustre: Skipped 1 previous similar message [12557.825772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12557.828418] Lustre: Skipped 1 previous similar message [12563.240591] Lustre: server umount lustre-MDT0001 complete [12565.165376] LustreError: 358434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12565.201136] LustreError: 358434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12576.552943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12576.666344] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12576.672273] Lustre: Found index 1 for lustre-MDT0001, updating log [12576.737748] Lustre: 358427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12577.264241] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12581.024302] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [12582.384551] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12584.981157] LustreError: 371386:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12584.985159] LustreError: 371386:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12585.008956] LustreError: 358429:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12585.023992] LustreError: 358429:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12585.114503] LustreError: 371389:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12585.134567] LustreError: 371389:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.121@tcp: cfg command failed: rc = -114 [12585.150506] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12585.150506] [12904.928769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12904.935895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12904.936589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12904.936594] Lustre: Skipped 3 previous similar messages [12904.976079] Lustre: Skipped 3 previous similar messages [12908.171061] Lustre: server umount lustre-MDT0000 complete [12910.052137] LustreError: 358438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12910.071933] LustreError: 358438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [12912.822302] LustreError: 358417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452797 with bad export cookie 6681812981923786114 [12912.829473] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12912.834544] LustreError: 358417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12915.168648] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12919.389567] Lustre: server umount lustre-MDT0001 complete [12925.067789] Lustre: server umount lustre-OST0000 complete [12935.861655] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [12947.627994] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [12996.109148] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 12:28:00 (1788452880) [13004.596348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13004.948750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13004.997897] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13005.157922] Lustre: lustre-MDT0000: new disk, initializing [13005.354562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13005.397456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13012.372876] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [13027.786994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13027.922830] Lustre: 381722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13027.966567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13027.969174] Lustre: Skipped 1 previous similar message [13028.052625] Lustre: lustre-MDT0001: new disk, initializing [13028.182745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13028.284069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13028.294390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13033.626642] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [13044.775341] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13051.850865] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13059.549974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13059.802347] Lustre: lustre-OST0000: new disk, initializing [13059.805643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13059.811758] Lustre: 383692:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13059.903591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13066.789979] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_default_debug -1 all [13068.956030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13068.996631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13069.138773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13079.069628] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13087.725097] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13087.730642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13087.747240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13087.755809] Lustre: Skipped 4 previous similar messages [13089.538271] Lustre: server umount lustre-MDT0000 complete [13089.765652] LustreError: 381733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13089.791593] LustreError: 381733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13094.606349] LustreError: 381714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452979 with bad export cookie 6681812981923788067 [13094.609514] LustreError: 382529:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13094.609822] LustreError: MGC192.168.201.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13094.613625] LustreError: 381714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13094.821900] Lustre: server umount lustre-MDT0001 complete [13110.189927] Lustre: server umount lustre-OST0000 complete [13144.801634] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing set_hostid [13157.543199] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing load_modules_local [13195.439466] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12926 sec ========= 12:31:19 (1788453079) [13197.666938] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:31:22 (1788453082) === [13202.419670] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:31:26 (1788453086) === [13251.124141] Lustre: DEBUG MARKER: oleg121-server.virtnet: executing unload_modules_local [13254.545781] Key type lgssc unregistered [13254.879756] LNet: 390063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13254.893138] LNetError: 390063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13254.907802] LNet: Removed LNI 192.168.201.121@tcp [13255.827598] Key type .llcrypt unregistered [13255.831068] Key type ._llcrypt unregistered