[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511871953 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003336] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009025] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014078] *** VALIDATE selinux *** [ 0.022802] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027655] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] *** VALIDATE tmpfs *** [ 0.031496] *** VALIDATE proc *** [ 0.032205] *** VALIDATE cgroup *** [ 0.032853] *** VALIDATE cgroup2 *** [ 0.034263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038003] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041818] debug: unmapping init [mem 0xffffffff9e459000-0xffffffff9e460fff] [ 0.043155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044650] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045025] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052259] rcu: Hierarchical SRCU implementation. [ 0.054510] smp: Bringing up secondary CPUs ... [ 0.055669] x86: Booting SMP configuration: [ 0.056037] .... node #0, CPUs: #1 #2 #3 [ 0.059235] smp: Brought up 1 node, 4 CPUs [ 0.061017] smpboot: Max logical packages: 1 [ 0.062016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141219] node 0 deferred pages initialised in 77ms [ 0.143125] devtmpfs: initialized [ 0.144255] x86/mm: Memory block size: 128MB [ 0.146942] gcov: version magic: 0x41383552 [ 0.148269] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150305] pinctrl core: initialized pinctrl subsystem [ 0.151158] [ 0.151597] ************************************************************* [ 0.152014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153014] ** ** [ 0.154015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155022] ** ** [ 0.156030] ** This means that this kernel is built to expose internal ** [ 0.157053] ** IOMMU data structures, which may compromise security on ** [ 0.158032] ** your system. ** [ 0.159016] ** ** [ 0.160016] ** If you see this message and you are not debugging the ** [ 0.161018] ** kernel, report this immediately to your vendor! ** [ 0.162019] ** ** [ 0.163019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164014] ************************************************************* [ 0.165991] NET: Registered protocol family 16 [ 0.167460] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170479] cpuidle: using governor menu [ 0.172538] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175461] PCI: Using configuration type 1 for base access [ 0.178135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186109] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188190] cryptd: max_cpu_qlen set to 1000 [ 0.191249] ACPI: Added _OSI(Module Device) [ 0.192015] ACPI: Added _OSI(Processor Device) [ 0.193028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198671] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201509] ACPI: Interpreter enabled [ 0.202067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203012] ACPI: Using IOAPIC for interrupt routing [ 0.204131] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.215946] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.216054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.217019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.218080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.220279] acpiphp: Slot [2] registered [ 0.221122] acpiphp: Slot [5] registered [ 0.221970] acpiphp: Slot [6] registered [ 0.222063] acpiphp: Slot [7] registered [ 0.223099] acpiphp: Slot [8] registered [ 0.223996] acpiphp: Slot [9] registered [ 0.224173] acpiphp: Slot [10] registered [ 0.225224] acpiphp: Slot [3] registered [ 0.226116] acpiphp: Slot [4] registered [ 0.227172] acpiphp: Slot [11] registered [ 0.228114] acpiphp: Slot [12] registered [ 0.229140] acpiphp: Slot [13] registered [ 0.230149] acpiphp: Slot [14] registered [ 0.231188] acpiphp: Slot [15] registered [ 0.232130] acpiphp: Slot [16] registered [ 0.233115] acpiphp: Slot [17] registered [ 0.234167] acpiphp: Slot [18] registered [ 0.235150] acpiphp: Slot [19] registered [ 0.236154] acpiphp: Slot [20] registered [ 0.237151] acpiphp: Slot [21] registered [ 0.238160] acpiphp: Slot [22] registered [ 0.239125] acpiphp: Slot [23] registered [ 0.240081] acpiphp: Slot [24] registered [ 0.241080] acpiphp: Slot [25] registered [ 0.242135] acpiphp: Slot [26] registered [ 0.243109] acpiphp: Slot [27] registered [ 0.245129] acpiphp: Slot [28] registered [ 0.246234] acpiphp: Slot [29] registered [ 0.248252] acpiphp: Slot [30] registered [ 0.249246] acpiphp: Slot [31] registered [ 0.251090] PCI host bridge to bus 0000:00 [ 0.252000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.254025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.258030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.261029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.264030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.267028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.270220] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.273103] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.276551] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.289096] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.295813] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.298020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.301025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.304021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.306884] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.310976] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.313048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.315746] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.320024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.334021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.339020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.344426] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.350015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.356036] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.375019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.387417] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.394000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.401012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.416031] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.429884] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.439023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.447017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.462018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.471721] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.480015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.486022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.507019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.520594] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.535030] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.549023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.579033] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.598567] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.619024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.640025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.657540] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.663025] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.667592] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.671553] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.675281] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.683044] iommu: Default domain type: Passthrough [ 0.684000] SCSI subsystem initialized [ 0.684000] ACPI: bus type USB registered [ 0.685113] usbcore: registered new interface driver usbfs [ 0.687064] usbcore: registered new interface driver hub [ 0.689087] usbcore: registered new device driver usb [ 0.691137] pps_core: LinuxPPS API ver. 1 registered [ 0.693012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.696066] PTP clock support registered [ 0.698202] EDAC MC: Ver: 3.0.0 [ 0.700179] PCI: Using ACPI for IRQ routing [ 0.704374] NetLabel: Initializing [ 0.705008] NetLabel: domain hash size = 128 [ 0.706010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.707070] NetLabel: unlabeled traffic allowed by default [ 0.710037] vgaarb: loaded [ 0.711219] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.712010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.721125] clocksource: Switched to clocksource kvm-clock [ 0.822099] VFS: Disk quotas dquot_6.6.0 [ 0.823559] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.826256] *** VALIDATE ramfs *** [ 0.827421] *** VALIDATE hugetlbfs *** [ 0.828910] pnp: PnP ACPI init [ 0.831133] pnp: PnP ACPI: found 6 devices [ 0.851818] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.855153] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.857408] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859902] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.862044] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.864599] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.867648] NET: Registered protocol family 2 [ 0.870160] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.874839] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.877792] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.883553] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.887252] TCP: Hash tables configured (established 65536 bind 65536) [ 0.890306] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.893425] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.896354] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.899646] NET: Registered protocol family 1 [ 0.903348] RPC: Registered named UNIX socket transport module. [ 0.905700] RPC: Registered udp transport module. [ 0.908369] RPC: Registered tcp transport module. [ 0.910575] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.913765] NET: Registered protocol family 44 [ 0.915458] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.917634] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.919667] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.921923] PCI: CLS 0 bytes, default 64 [ 0.923560] Unpacking initramfs... [ 2.312442] debug: unmapping init [mem 0xffff99c63cc54000-0xffff99c63ffbffff] [ 2.315177] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.316605] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.318631] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.795581] Initialise system trusted keyrings [ 2.796705] Key type blacklist registered [ 2.798122] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.808228] zbud: loaded [ 2.812276] *** VALIDATE nfs *** [ 2.813550] *** VALIDATE nfs4 *** [ 2.815290] pstore: using deflate compression [ 2.818425] Platform Keyring initialized [ 2.897955] NET: Registered protocol family 38 [ 2.899658] Key type asymmetric registered [ 2.901313] Asymmetric key parser 'x509' registered [ 2.902836] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.907628] io scheduler mq-deadline registered [ 2.909542] io scheduler kyber registered [ 2.911307] io scheduler bfq registered [ 2.913425] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.916713] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.920239] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.923312] ACPI: Power Button [PWRF] [ 2.928216] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.935342] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.948820] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.959782] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.988168] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.017776] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.059795] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.064132] Non-volatile memory driver v1.3 [ 3.065782] Linux agpgart interface v0.103 [ 3.103193] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.106602] vda: detected capacity change from 0 to 74690560 [ 3.123629] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.126592] vdb: detected capacity change from 0 to 1073741824 [ 3.140321] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.142552] vdc: detected capacity change from 0 to 2621440000 [ 3.159956] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.163474] vdd: detected capacity change from 0 to 2621440000 [ 3.181270] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.184198] vde: detected capacity change from 0 to 4294967296 [ 3.199182] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.201629] vdf: detected capacity change from 0 to 4294967296 [ 3.207434] libphy: Fixed MDIO Bus: probed [ 3.227207] usbcore: registered new interface driver usbserial_generic [ 3.229856] usbserial: USB Serial support registered for generic [ 3.232343] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.237052] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.238913] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.241602] mousedev: PS/2 mouse device common for all mice [ 3.244720] rtc_cmos 00:05: RTC can wake from S4 [ 3.244828] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.250339] rtc_cmos 00:05: registered as rtc0 [ 3.254463] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.257873] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.261419] intel_pstate: CPU model not supported [ 3.268493] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.274891] hid: raw HID events driver (C) Jiri Kosina [ 3.280235] usbcore: registered new interface driver usbhid [ 3.284585] usbhid: USB HID core driver [ 3.288644] drop_monitor: Initializing network drop monitor service [ 3.294599] Initializing XFRM netlink socket [ 3.296803] NET: Registered protocol family 10 [ 3.300171] Segment Routing with IPv6 [ 3.301628] NET: Registered protocol family 17 [ 3.304029] mpls_gso: MPLS GSO support [ 3.311488] RAS: Correctable Errors collector initialized. [ 3.313839] AVX version of gcm_enc/dec engaged. [ 3.315674] AES CTR mode by8 optimization enabled [ 3.403417] sched_clock: Marking stable (3403393786, 0)->(4384187137, -980793351) [ 3.407459] registered taskstats version 1 [ 3.411220] Loading compiled-in X.509 certificates [ 3.414534] zswap: loaded using pool lzo/zbud [ 3.444268] Key type big_key registered [ 3.458651] Key type encrypted registered [ 3.460389] ima: No TPM chip found, activating TPM-bypass! [ 3.463110] ima: Allocated hash algorithm: sha1 [ 3.465995] ima: No architecture policies found [ 3.468381] evm: Initialising EVM extended attributes: [ 3.470539] evm: security.selinux [ 3.472242] evm: security.ima [ 3.473648] evm: security.capability [ 3.475092] evm: HMAC attrs: 0x1 [ 3.478543] rtc_cmos 00:05: setting system clock to 2026-09-10 23:22:05 UTC (1789082525) [ 3.485207] debug: unmapping init [mem 0xffffffff9f403000-0xffffffff9f5fffff] [ 3.487954] debug: unmapping init [mem 0xffffffff9e182000-0xffffffff9e458fff] [ 3.496114] Write protecting the kernel read-only data: 28672k [ 3.499789] debug: unmapping init [mem 0xffffffff9c803000-0xffffffff9c9fffff] [ 3.503318] debug: unmapping init [mem 0xffffffff9d114000-0xffffffff9d1fffff] [ 3.547248] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.557370] systemd[1]: Detected virtualization kvm. [ 3.558850] systemd[1]: Detected architecture x86-64. [ 3.560340] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.586957] systemd[1]: No hostname configured. [ 3.589759] systemd[1]: Set hostname to . [ 3.593389] random: systemd: uninitialized urandom read (16 bytes read) [ 3.597183] systemd[1]: Initializing machine ID from random generator. [ 3.647780] random: ln: uninitialized urandom read (6 bytes read) [ 3.745147] random: systemd: uninitialized urandom read (16 bytes read) [ 3.748912] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.754561] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.759687] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.385695] device-mapper: uevent: version 1.0.3 [ 4.387564] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.102006] random: fast init done [ 5.104481] virtio_net virtio0 ens2: renamed from eth0 [ 5.182761] scsi host0: ata_piix [ 5.188078] scsi host1: ata_piix [ 5.189754] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.192811] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.757332] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.986828] random: crng init done [ 9.990661] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.077235] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.558302] printk: systemd: 25 output lines suppressed due to ratelimiting [ 15.497766] SELinux: Disabled at runtime. [ 15.638310] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.665077] systemd[1]: Detected virtualization kvm. [ 15.674343] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.169668] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.176761] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.187250] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.191605] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.199146] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.211639] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.241445] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket.[ 17.622427] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.919541] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.253878] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.571591] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.364342] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.530682] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.819441] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 26.852849] NFS: Registering the id_resolver key type [ 26.858148] Key type id_resolver registered [ 26.878470] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg252-server login: [ 42.691022] hrtimer: interrupt took 25950868 ns [ 92.860535] libcfs: loading out-of-tree module taints kernel. [ 92.958323] Key type ._llcrypt registered [ 92.959989] Key type .llcrypt registered [ 93.122296] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 118.907850] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 120.951554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 120.964677] alg: No test for adler32 (adler32-zlib) [ 122.752961] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 124.311042] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 126.202358] Key type lgssc registered [ 128.005197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.226685] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 198.071827] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 213.496749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 213.530280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.989663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 215.041458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 215.255631] Lustre: lustre-MDT0000: new disk, initializing [ 215.392321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.414420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.933185] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.673369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.812629] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 235.849708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 235.855435] Lustre: Skipped 1 previous similar message [ 235.957696] Lustre: lustre-MDT0001: new disk, initializing [ 236.042340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.083995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 236.097166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 241.117436] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.109694] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 257.015299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 257.388515] Lustre: lustre-OST0000: new disk, initializing [ 257.394964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 257.415536] Lustre: 8449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 257.494965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.296421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 258.307861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 258.404415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 265.406380] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.255469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 283.459460] Lustre: lustre-OST0001: new disk, initializing [ 283.469457] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 283.473507] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 283.549144] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 291.659508] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 292.445770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 292.456164] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 292.596646] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 305.407672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 314.112339] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 321.784164] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing check_logdir /tmp/testlogs/ [ 327.474698] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing yml_node [ 332.690541] Lustre: DEBUG MARKER: Client: 2.17.55.25 [ 335.803799] Lustre: DEBUG MARKER: MDS: 2.17.55.25 [ 339.464887] Lustre: DEBUG MARKER: OSS: 2.17.55.25 [ 341.623372] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 19:27:40 EDT 2026 [ 364.548957] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 366.397474] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 384.484329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 384.490677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.500408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 384.995151] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.014285] Lustre: Skipped 2 previous similar messages [ 385.024813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.032374] Lustre: Skipped 2 previous similar messages [ 388.633790] Lustre: server umount lustre-MDT0000 complete [ 392.814242] LustreError: 7470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789082914 with bad export cookie 1383372162624761159 [ 392.821105] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 392.824573] LustreError: 7470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 395.236444] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.237059] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 395.252464] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 395.283164] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 395.293989] Lustre: Skipped 1 previous similar message [ 397.280533] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 397.296132] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 399.391721] Lustre: server umount lustre-MDT0001 complete [ 403.790899] Lustre: server umount lustre-OST0000 complete [ 407.475831] Lustre: server umount lustre-OST0001 complete [ 416.017402] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 427.399495] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 476.350596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.739378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 476.824744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 476.988182] Lustre: lustre-MDT0000: new disk, initializing [ 477.131908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.172874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 483.107092] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 496.047837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.216829] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 496.289919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 496.292735] Lustre: Skipped 1 previous similar message [ 496.346792] Lustre: lustre-MDT0001: new disk, initializing [ 496.581245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 496.700895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 496.734069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 503.375470] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 516.119696] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 524.014605] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 532.718206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 532.860914] Lustre: lustre-OST0000: new disk, initializing [ 532.863213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 532.870681] Lustre: 17049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 532.933866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.415293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 534.426182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 534.544116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 540.540445] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 551.484283] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 556.277369] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 556.522819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 561.899774] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 562.187405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 565.216115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 565.234937] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.240290] LustreError: Skipped 1 previous similar message [ 565.253996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 565.273403] Lustre: Skipped 2 previous similar messages [ 565.305414] Lustre: Skipped 3 previous similar messages [ 570.346261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 570.351038] Lustre: Skipped 1 previous similar message [ 570.684246] Lustre: server umount lustre-OST0000 complete [ 575.458248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.464973] Lustre: Skipped 1 previous similar message [ 580.579160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 580.590560] Lustre: Skipped 3 previous similar messages [ 589.279653] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 589.816755] Lustre: server umount lustre-MDT0000 complete [ 590.836466] LustreError: 15090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.869575] LustreError: 15090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 595.964743] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.997278] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 596.005316] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083118 with bad export cookie 1383372162624762692 [ 596.007801] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.021698] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 596.786410] Lustre: server umount lustre-MDT0001 complete [ 604.005969] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 19:32:02 (1789083122) [ 611.206269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.564944] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.631780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 616.720524] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 621.029821] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.082775] LustreError: 19319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 627.770520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.407295] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 642.921678] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 649.258966] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 656.019373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 656.297236] LustreError: 21139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.322653] LustreError: 21139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 663.492813] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 671.220899] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 678.071373] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 689.131046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.135864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 689.145158] Lustre: Skipped 2 previous similar messages [ 689.161752] Lustre: Skipped 4 previous similar messages [ 699.360325] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 699.693749] Lustre: server umount lustre-MDT0000 complete [ 703.870207] LustreError: 19300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083225 with bad export cookie 1383372162624763574 [ 703.887743] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.275602] Lustre: server umount lustre-MDT0001 complete [ 719.601514] Lustre: server umount lustre-OST0000 complete [ 744.035996] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 757.512782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.104128] LustreError: 23705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 758.225173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.232825] Lustre: Skipped 2 previous similar messages [ 764.165160] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 775.510902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.243991] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 785.764780] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 793.762175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 802.327738] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 812.583982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 812.837324] Lustre: lustre-OST0001: new disk, initializing [ 812.841375] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 812.849338] Lustre: 26023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 814.268885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 814.286902] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 814.487975] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 821.012921] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 831.994799] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 836.893091] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 850.415433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.425811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 850.441764] Lustre: Skipped 3 previous similar messages [ 850.456428] Lustre: Skipped 9 previous similar messages [ 855.912217] Lustre: server umount lustre-MDT0000 complete [ 860.358304] LustreError: 23686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083382 with bad export cookie 1383372162624764421 [ 860.378608] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 860.385822] LustreError: 23686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 860.642351] LustreError: 23702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.652520] LustreError: 23702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 860.856319] Lustre: server umount lustre-MDT0001 complete [ 867.179308] Lustre: server umount lustre-OST0000 complete [ 871.674845] Lustre: server umount lustre-OST0001 complete [ 894.215510] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 906.699697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.423526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.440821] Lustre: Skipped 3 previous similar messages [ 913.676306] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 923.936837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.206370] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 933.575339] Lustre: 30189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 940.461439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 946.771492] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 956.187559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 964.503327] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 973.432314] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 977.729433] Lustre: 32070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 986.085095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 986.101089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.120533] Lustre: Skipped 2 previous similar messages [ 986.123676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.138438] Lustre: Skipped 6 previous similar messages [ 990.895585] Lustre: server umount lustre-MDT0000 complete [ 992.742278] LustreError: 29037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 992.768762] LustreError: 29037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 994.832688] LustreError: 31381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083516 with bad export cookie 1383372162624765716 [ 994.844530] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 994.858945] LustreError: 31381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 995.174693] Lustre: server umount lustre-MDT0001 complete [ 1010.420954] Lustre: server umount lustre-OST0000 complete [ 1013.729949] Lustre: 3645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789083519/real 1789083519] req@ffff99c587429180 x1875989124922112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789083535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1015.264964] Lustre: 3648:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789083521/real 1789083521] req@ffff99c587428380 x1875989124922368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789083537 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1015.551286] Lustre: server umount lustre-OST0001 complete [ 1028.989616] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 19:39:07 (1789083547) [ 1061.214578] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1075.477549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.120178] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1088.580279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.930291] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1097.564794] Lustre: 36024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1104.268969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1111.432377] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1121.180220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1129.313400] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1137.325762] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1141.412706] Lustre: 37903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1150.433257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1150.442748] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.448374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1150.448395] Lustre: Skipped 4 previous similar messages [ 1150.500711] Lustre: Skipped 6 previous similar messages [ 1152.770945] Lustre: server umount lustre-MDT0000 complete [ 1156.679013] LustreError: 34855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083678 with bad export cookie 1383372162624766822 [ 1156.681360] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1156.704755] LustreError: 34855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1157.222227] Lustre: server umount lustre-MDT0001 complete [ 1171.419641] Lustre: server umount lustre-OST0000 complete [ 1185.764735] Lustre: server umount lustre-OST0001 complete [ 1197.252735] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 19:41:55 (1789083715) [ 1216.103767] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1227.337246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1227.878920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.889967] Lustre: Skipped 7 previous similar messages [ 1232.950242] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1243.191043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.399800] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1251.020257] Lustre: 41192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1257.668371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1264.398634] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1268.405145] LustreError: 41546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.423490] LustreError: 41546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 1275.409575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.207455] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1292.995347] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1296.653300] Lustre: 43072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1297.765867] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1297.768838] Lustre: Skipped 1 previous similar message [ 1301.490137] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.515546] Lustre: Skipped 3 previous similar messages [ 1306.759813] Lustre: server umount lustre-MDT0000 complete [ 1310.814032] LustreError: 41575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083832 with bad export cookie 1383372162624767928 [ 1310.824365] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1310.825322] LustreError: 41575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1317.497947] Lustre: server umount lustre-MDT0001 complete [ 1327.627029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.043389] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1345.002512] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1350.175596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.729969] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1356.286815] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1356.291835] Lustre: Skipped 1 previous similar message [ 1363.167515] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1364.869506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1371.154935] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1372.787680] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1381.344836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1384.555711] Lustre: server umount lustre-MDT0000 complete [ 1388.738694] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789083910 with bad export cookie 1383372162624769062 [ 1388.748556] LustreError: 40025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1388.748631] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1389.212656] Lustre: server umount lustre-MDT0001 complete [ 1404.057371] Lustre: server umount lustre-OST0000 complete [ 1408.480685] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789083914/real 1789083914] req@ffff99c687734e00 x1875989125059584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789083930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1408.982589] Lustre: server umount lustre-OST0001 complete [ 1419.456612] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 19:45:38 (1789083938) [ 1438.708408] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1452.088830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.268462] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1468.867051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.092607] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1478.680157] Lustre: 48837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1485.702419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1493.144274] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1502.124743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1509.756570] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1518.250611] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1523.061663] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1535.977352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1535.991293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1535.999541] Lustre: Skipped 21 previous similar messages [ 1537.867182] Lustre: server umount lustre-MDT0000 complete [ 1542.063744] LustreError: 47665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084064 with bad export cookie 1383372162624769853 [ 1542.081892] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1542.552302] Lustre: server umount lustre-MDT0001 complete [ 1557.376422] Lustre: server umount lustre-OST0000 complete [ 1559.522339] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789084065/real 1789084065] req@ffff99c58776f100 x1875989125116800/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789084081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1559.566118] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1559.601793] Lustre: Skipped 12 previous similar messages [ 1562.610967] Lustre: server umount lustre-OST0001 complete [ 1576.199052] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 19:48:14 (1789084094) [ 1579.364917] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1581.339973] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 19:48:20 (1789084100) [ 1592.638757] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 1602.413536] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1627.132914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.451834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1627.544971] Lustre: lustre-MDT0000: new disk, initializing [ 1627.657616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1633.194372] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1643.536044] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1651.648946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1652.066769] Lustre: lustre-OST0000: new disk, initializing [ 1652.072157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1652.075133] Lustre: Skipped 1 previous similar message [ 1652.082407] Lustre: 54914:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1653.231056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1653.240279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1653.272452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1659.373539] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1670.168598] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1673.557161] Lustre: DEBUG MARKER: create test files [ 1717.239619] Lustre: server umount lustre-OST0000 complete [ 1721.253350] Lustre: server umount lustre-MDT0000 complete [ 1734.213773] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1738.630499] Key type lgssc unregistered [ 1739.118448] LNet: 56544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.125744] LNetError: 56544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.212911] LNet: Removed LNI 192.168.202.152@tcp [ 1741.631169] Key type .llcrypt unregistered [ 1741.633833] Key type ._llcrypt unregistered [ 1774.915396] Key type ._llcrypt registered [ 1774.919659] Key type .llcrypt registered [ 1775.001720] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1776.549182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.595105] alg: No test for adler32 (adler32-zlib) [ 1777.702856] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 1778.062390] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1779.743220] Key type lgssc registered [ 1780.978885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1789.727319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1789.748861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.213159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1796.530964] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1804.710950] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1812.447835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1812.763927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.879807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1817.000849] Lustre: 59281:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1817.015734] Lustre: 59281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1817.034018] Lustre: 59281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1817.045756] Lustre: 59281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1817.062928] Lustre: 59281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1817.087687] Lustre: 59281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1820.777310] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1831.644801] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1838.520885] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1844.628892] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1852.896186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1852.904787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.916902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1858.023137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1863.147460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.231239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1867.402643] Lustre: server umount lustre-OST0000 complete [ 1871.616869] Lustre: server umount lustre-MDT0000 complete [ 1886.452843] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 1890.435607] Key type lgssc unregistered [ 1890.726875] LNet: 60892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.733822] LNetError: 60892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.755316] LNet: Removed LNI 192.168.202.152@tcp [ 1891.537914] Key type .llcrypt unregistered [ 1891.539572] Key type ._llcrypt unregistered [ 1923.232258] Key type ._llcrypt registered [ 1923.234395] Key type .llcrypt registered [ 1923.342431] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 1924.951026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1924.965434] alg: No test for adler32 (adler32-zlib) [ 1926.031124] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 1926.269539] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 1927.985222] Key type lgssc registered [ 1929.340374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1938.146640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1938.163310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.757768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.947623] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1953.438200] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1960.705239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1961.059164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.144159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1964.208557] Lustre: 63728:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1964.211340] Lustre: 63728:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1964.213498] Lustre: 63728:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1964.216306] Lustre: 63728:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1964.219120] Lustre: 63728:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1964.222342] Lustre: 63728:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1969.140649] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 1979.660189] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1985.524094] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 2000.366332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2000.368863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.374740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.501229] Lustre: server umount lustre-OST0000 complete [ 2007.573620] Lustre: server umount lustre-MDT0000 complete [ 2019.975814] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 2022.742250] Key type lgssc unregistered [ 2023.044628] LNet: 65289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.049652] LNetError: 65289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.067126] LNet: Removed LNI 192.168.202.152@tcp [ 2023.977168] Key type .llcrypt unregistered [ 2023.980739] Key type ._llcrypt unregistered [ 2048.413037] Key type ._llcrypt registered [ 2048.418721] Key type .llcrypt registered [ 2048.571247] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 2064.414481] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2065.875239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2066.009174] alg: No test for adler32 (adler32-zlib) [ 2067.162778] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 2067.336677] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 2068.975262] Key type lgssc registered [ 2070.043203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2115.641292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2115.673053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.022319] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2117.050314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2117.174158] Lustre: lustre-MDT0000: new disk, initializing [ 2117.310841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.343616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2122.677388] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2136.175345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.365980] Lustre: 68444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2136.421744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2136.424922] Lustre: Skipped 1 previous similar message [ 2136.518233] Lustre: lustre-MDT0001: new disk, initializing [ 2136.607979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.641480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2136.657639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2142.132513] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2153.604168] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2159.930240] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2168.668502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2168.992926] Lustre: lustre-OST0000: new disk, initializing [ 2168.995643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2169.000808] Lustre: 70412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2169.108374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2177.182234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2177.209877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2177.460434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2177.890440] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2189.293192] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2194.370578] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2194.605921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2199.680668] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2199.866343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2203.105245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.111262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.122964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.136377] Lustre: Skipped 1 previous similar message [ 2208.053741] Lustre: server umount lustre-OST0000 complete [ 2213.344313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.348034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.365536] Lustre: Skipped 1 previous similar message [ 2213.381828] Lustre: Skipped 2 previous similar messages [ 2218.464769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2218.480807] Lustre: Skipped 1 previous similar message [ 2218.812180] Lustre: server umount lustre-MDT0000 complete [ 2223.248062] LustreError: 70411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084745 with bad export cookie 9077813413821652434 [ 2223.253417] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2223.273471] LustreError: 70411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2223.662445] Lustre: server umount lustre-MDT0001 complete [ 2234.536683] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 19:59:13 (1789084753) [ 2246.473849] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2291.057594] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 2303.169767] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2347.076587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.446100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2347.496572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2347.592821] Lustre: lustre-MDT0000: new disk, initializing [ 2347.702478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.731396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2352.967308] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2365.260713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.421991] Lustre: 76453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2365.513314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2365.518213] Lustre: Skipped 1 previous similar message [ 2365.630424] Lustre: lustre-MDT0001: new disk, initializing [ 2365.818489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2365.879755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2365.898624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2370.761581] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2380.365043] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2387.481607] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2395.718476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.188480] Lustre: lustre-OST0000: new disk, initializing [ 2396.193185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2396.204766] Lustre: 78420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2396.372083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.304182] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2404.412046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2404.431567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2404.611129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2414.237617] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2420.163617] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2420.630756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2426.623939] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2427.086814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2430.434461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.436313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.451830] LustreError: Skipped 2 previous similar messages [ 2430.453145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.510428] Lustre: Skipped 1 previous similar message [ 2435.563052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.570437] Lustre: Skipped 2 previous similar messages [ 2444.255190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2444.708848] Lustre: server umount lustre-OST0000 complete [ 2450.912899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.922344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.925637] Lustre: Skipped 1 previous similar message [ 2450.936735] Lustre: Skipped 3 previous similar messages [ 2455.967138] Lustre: server umount lustre-MDT0000 complete [ 2459.990962] LustreError: 76444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789084982 with bad export cookie 9077813413821653316 [ 2460.014360] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.320357] Lustre: server umount lustre-MDT0001 complete [ 2469.884727] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 20:03:08 (1789084988) [ 2477.354905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.871409] LustreError: 80835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2477.890486] LustreError: 80835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2477.961250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.557359] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2483.184479] LustreError: 80836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.274706] LustreError: 80835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.387646] LustreError: 80836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2492.809301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.578850] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2508.713883] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2515.945205] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2524.121067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.626919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.632662] Lustre: Skipped 1 previous similar message [ 2532.375973] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2540.860535] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2545.485130] Lustre: 83510:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 90175d1f-639b-4c12-9a2c-5966543797de at adminstrative request [ 2547.176153] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2547.199550] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2557.414692] Lustre: MGS: Client 90175d1f-639b-4c12-9a2c-5966543797de (at 0@lo) reconnecting [ 2557.428348] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x7dfadf341c1da4b6 to 0x7dfadf341c1da756 [ 2557.447997] Lustre: MGC192.168.202.152@tcp: Connection restored to 0@lo (at 0@lo) [ 2586.504678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2586.739692] Lustre: lustre-OST0001: new disk, initializing [ 2586.742941] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2586.747722] Lustre: 83904:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2592.926942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2592.953321] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2593.151908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2594.669025] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2606.902584] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2613.739082] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2613.739687] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2613.755615] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2613.758419] Lustre: Skipped 1 previous similar message [ 2617.214268] Lustre: server umount lustre-OST0001 complete [ 2618.861789] LustreError: 82661:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.891213] LustreError: 82661:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2623.983989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2624.004808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2624.014060] LustreError: Skipped 1 previous similar message [ 2624.042692] Lustre: Skipped 2 previous similar messages [ 2627.525489] Lustre: server umount lustre-OST0000 complete [ 2638.938336] Lustre: server umount lustre-MDT0000 complete [ 2639.328585] LustreError: 80832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.342341] LustreError: 80832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2643.245044] LustreError: 81571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789085165 with bad export cookie 9077813413821654870 [ 2643.255835] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.262917] LustreError: 81571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2643.670616] Lustre: server umount lustre-MDT0001 complete [ 2653.800446] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 20:06:12 (1789085172) [ 2655.618275] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2657.480079] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 20:06:16 (1789085176) [ 2659.107927] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2661.092723] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 20:06:20 (1789085180) [ 2662.947774] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2664.683444] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 20:06:23 (1789085183) [ 2666.612441] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2668.731718] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 20:06:27 (1789085187) [ 2670.613379] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2676.939895] loop: module loaded [ 2676.957072] loop0: detected capacity change from 0 to 4294967296 [ 2677.032685] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2679.057715] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2682.568187] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2687.183548] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2717.587031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2717.644556] LustreError: 88028:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2717.661823] LustreError: 88028:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2717.677183] LustreError: 88028:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2717.693918] LustreError: 88028:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2740.284801] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 2752.478821] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 2794.382473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.722542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2794.781228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2794.887712] Lustre: lustre-MDT0000: new disk, initializing [ 2794.992189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.002635] Lustre: Skipped 1 previous similar message [ 2795.029929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2799.602687] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2813.983914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.093899] Lustre: 90895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2814.692317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2819.708807] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2830.353893] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2836.206811] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2845.114689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2845.538844] Lustre: 92865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2852.594173] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2853.488786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2853.623070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2864.026637] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2871.081810] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2871.521237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2876.509292] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2876.705256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2879.474350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2879.475506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.476626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2879.476631] Lustre: Skipped 4 previous similar messages [ 2879.532837] Lustre: Skipped 3 previous similar messages [ 2893.279271] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2893.553699] Lustre: server umount lustre-OST0000 complete [ 2905.389866] Lustre: server umount lustre-MDT0000 complete [ 2910.186177] LustreError: 90901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2910.217552] LustreError: 90901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2910.410729] LustreError: 92864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789085432 with bad export cookie 9077813413821655486 [ 2910.416605] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.434843] LustreError: 92864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2910.796676] Lustre: server umount lustre-MDT0001 complete [ 2920.471690] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 20:10:39 (1789085439) [ 2928.059520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.543188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2928.548407] Lustre: Skipped 2 previous similar messages [ 2933.380904] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2942.722462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.416359] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2956.253923] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2962.654323] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2969.755330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.712878] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 2983.868742] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2988.305509] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2988.558367] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2993.112657] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2993.362075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2999.940556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3000.133590] Lustre: lustre-OST0001: new disk, initializing [ 3000.136391] Lustre: Skipped 2 previous similar messages [ 3000.138806] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3000.141057] Lustre: Skipped 3 previous similar messages [ 3000.143438] Lustre: 98652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3006.953183] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3007.065241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3007.072171] Lustre: Skipped 2 previous similar messages [ 3007.083781] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3007.277110] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3018.429652] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3024.277965] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3024.527927] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3029.944418] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3030.144366] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3049.629927] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3075.487163] Lustre: Failing over lustre-MDT0000 [ 3076.075251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3076.085891] Lustre: server umount lustre-MDT0000 complete [ 3076.087356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.087363] Lustre: Skipped 2 previous similar messages [ 3076.089169] LustreError: 95282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.141688] LustreError: 95282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3095.522342] Lustre: 66666:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789085601/real 1789085601] req@0000000040fa97b5 x1875991162285440/t0(0) o400->MGC192.168.202.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789085617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3095.559336] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3101.641232] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3101.647665] LDISKFS-fs (dm-0): recovery complete [ 3101.660328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.850996] LustreError: 66664:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005ef55a11 x1875991162294784/t0(0) o250->MGC192.168.202.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3107.178163] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3109.081095] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3109.082804] Lustre: 97122:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3112.189782] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3112.419264] Lustre: 96009:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3112.429283] Lustre: 96009:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 1 previous similar message [ 3112.492842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3112.604786] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3112.943438] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3113.268569] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3113.279110] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 3113.607081] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3113.620689] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 1 previous similar message [ 3113.632977] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3113.657714] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 7 previous similar messages [ 3114.294062] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3114.314668] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 2 previous similar messages [ 3114.647130] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3114.652714] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 2 previous similar messages [ 3115.930059] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3115.969777] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 6 previous similar messages [ 3116.313178] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3116.332530] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 5 previous similar messages [ 3116.655990] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3116.673164] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 5 previous similar messages [ 3120.229358] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3120.264681] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 12 previous similar messages [ 3120.608344] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3120.621768] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 3120.935237] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3120.940628] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 12 previous similar messages [ 3128.269597] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3128.296155] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 24 previous similar messages [ 3128.616815] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3128.633754] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 24 previous similar messages [ 3128.959198] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3128.964107] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 24 previous similar messages [ 3144.560202] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3144.590139] Lustre: 101499:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 50 previous similar messages [ 3144.921811] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3144.931765] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 50 previous similar messages [ 3145.263152] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3145.269923] LustreError: 101499:0:(ldlm_lib.c:2697:replay_request_or_update()) Skipped 50 previous similar messages [ 3169.295159] Lustre: 101499:0:(ldlm_lib.c:2393:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3169.307457] Lustre: 101499:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3169.320128] Lustre: 101499:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 406adc56-95e0-405e-a87f-042ede7acca0@192.168.202.52@tcp [ 3169.338078] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3169.343429] Lustre: 101499:0:(ldlm_lib.c:1913:abort_req_replay_queue()) @@@ aborted: req@00000000113a11e2 x1875991144239872/t0(8589934774) o35->406adc56-95e0-405e-a87f-042ede7acca0@192.168.202.52@tcp:707/0 lens 392/0 e 0 to 0 dl 1789085702 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3169.389194] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3169.404379] Lustre: lustre-MDT0000: Denying connection for new client 406adc56-95e0-405e-a87f-042ede7acca0 (at 192.168.202.52@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3169.427280] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3169.478602] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3169.482522] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3169.509241] Lustre: Skipped 2 previous similar messages [ 3169.536642] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:46 to 0x2c0000401:545) [ 3169.537739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 3188.205981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3188.216990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.225023] Lustre: Skipped 3 previous similar messages [ 3188.227976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3188.232492] Lustre: Skipped 9 previous similar messages [ 3192.736790] Lustre: server umount lustre-OST0000 complete [ 3194.344365] LustreError: 97104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.378443] LustreError: 97104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 45 previous similar messages [ 3211.231406] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3211.364215] Lustre: server umount lustre-OST0001 complete [ 3230.175963] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3230.325772] Lustre: server umount lustre-MDT0000 complete [ 3234.557948] LustreError: 95263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789085756 with bad export cookie 9077813413821706705 [ 3234.559768] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3234.567038] LustreError: 95263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3234.984082] Lustre: server umount lustre-MDT0001 complete [ 3245.621361] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 20:16:04 (1789085764) [ 3254.076585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3254.082586] Lustre: *** cfs_fail_loc=197, val=0*** [ 3269.602709] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3274.719680] LustreError: 103477:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [ 3279.859402] Lustre: *** cfs_fail_loc=197, val=0*** [ 3279.863931] Lustre: Skipped 65 previous similar messages [ 3300.450057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3300.460639] Lustre: Skipped 4 previous similar messages [ 3308.434410] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3317.665590] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3320.649652] Lustre: server umount lustre-OST0000 complete [ 3331.165994] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 20:17:29 (1789085849) [ 3354.035371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.340687] LustreError: 105099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.356896] LustreError: 105099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3359.684843] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3372.914230] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3378.791480] LustreError: 105130:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3378.811259] LustreError: 105130:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3378.825799] LustreError: 105130:0:(lod_dev.c:512:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 3379.387493] Lustre: server umount lustre-MDT0000 complete [ 3419.114407] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3431.118305] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3474.083027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.329709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3474.353244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3474.437178] Lustre: lustre-MDT0000: new disk, initializing [ 3474.515393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3480.163632] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3492.925948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.109793] Lustre: 109308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3493.276936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3498.568575] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3509.713287] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3515.711990] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3522.945534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3523.270539] Lustre: 111279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3530.285020] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3533.038050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3541.080309] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3546.468353] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3546.811673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3551.144356] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3551.352394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3553.760610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3553.770817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.773688] LustreError: Skipped 2 previous similar messages [ 3553.776578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.798775] Lustre: Skipped 6 previous similar messages [ 3553.834823] Lustre: Skipped 16 previous similar messages [ 3559.825631] Lustre: server umount lustre-OST0000 complete [ 3578.335308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3578.520564] Lustre: server umount lustre-MDT0000 complete [ 3582.253280] LustreError: 111278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789086104 with bad export cookie 9077813413821717338 [ 3582.256113] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3582.267077] LustreError: 111278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3582.709907] Lustre: server umount lustre-MDT0001 complete [ 3593.023696] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 20:21:52 (1789086112) [ 3602.908858] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 3605.813352] Key type lgssc unregistered [ 3606.208580] LNet: 113767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3606.220506] LNetError: 113767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3606.247266] LNet: Removed LNI 192.168.202.152@tcp [ 3607.296925] Key type .llcrypt unregistered [ 3607.299247] Key type ._llcrypt unregistered [ 3627.727430] Key type ._llcrypt registered [ 3627.732355] Key type .llcrypt registered [ 3627.929871] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3644.146794] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3645.887054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3646.178694] alg: No test for adler32 (adler32-zlib) [ 3647.231594] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 3647.461440] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 3649.272159] Key type lgssc registered [ 3650.571279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3698.554117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3698.653594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.985136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3700.047388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3700.168287] Lustre: lustre-MDT0000: new disk, initializing [ 3700.258820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3700.279145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3704.811320] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3715.663267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3715.948541] Lustre: lustre-OST0000: new disk, initializing [ 3715.951388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3715.957382] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3715.963357] Lustre: Skipped 1 previous similar message [ 3715.970143] Lustre: 117949:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3716.105184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3721.286071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3721.313789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3721.365954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3722.802293] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3733.470695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.686618] Lustre: lustre-OST0001: new disk, initializing [ 3733.690532] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3733.698427] Lustre: 118810:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3733.786087] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3739.783740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3739.789254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3739.803992] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3740.556232] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3754.985409] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.990414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3755.014580] Lustre: Skipped 1 previous similar message [ 3756.365454] Lustre: server umount lustre-MDT0000 complete [ 3763.426733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3767.665682] LustreError: 117134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789086289 with bad export cookie 12264536830036446293 [ 3767.683477] LustreError: 117134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3767.684131] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3767.811166] Lustre: server umount lustre-OST0000 complete [ 3771.315619] Lustre: server umount lustre-OST0001 complete [ 3781.613825] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 20:25:00 (1789086300) [ 3805.146919] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 20:25:24 (1789086324) [ 3816.459616] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3841.687690] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 20:26:00 (1789086360) [ 3887.374305] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 3897.413504] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 3936.188645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.469779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3936.495898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3936.564609] Lustre: lustre-MDT0000: new disk, initializing [ 3936.639076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3936.658440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3942.054334] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3954.748330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3954.895455] Lustre: 126567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3954.929184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3954.932461] Lustre: Skipped 1 previous similar message [ 3954.989234] Lustre: lustre-MDT0001: new disk, initializing [ 3955.134628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3955.214444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3955.238404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3959.762056] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3969.571513] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3976.930932] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3985.436583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3985.791673] Lustre: lustre-OST0000: new disk, initializing [ 3985.805779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3985.821682] Lustre: 128534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3985.988470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3994.928391] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 3995.783111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3995.794357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3995.930775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4006.048390] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4012.226849] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4012.603922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4018.977899] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4019.214326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4021.734977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4021.742185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.743744] LustreError: Skipped 1 previous similar message [ 4021.744656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.744661] Lustre: Skipped 1 previous similar message [ 4021.760768] Lustre: Skipped 1 previous similar message [ 4026.854569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.861732] Lustre: Skipped 2 previous similar messages [ 4028.095118] Lustre: server umount lustre-OST0000 complete [ 4037.106117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4037.112401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.126634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4037.150284] Lustre: Skipped 1 previous similar message [ 4037.168733] Lustre: Skipped 1 previous similar message [ 4039.813441] Lustre: server umount lustre-MDT0000 complete [ 4043.969866] LustreError: 126558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789086565 with bad export cookie 12264536830036447518 [ 4043.979527] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4043.995064] LustreError: 126558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4044.400964] Lustre: server umount lustre-MDT0001 complete [ 4054.643378] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 20:29:33 (1789086573) [ 4062.767550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4063.271507] LustreError: 130948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.284422] LustreError: 130948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4063.333561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.488780] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4068.847549] LustreError: 130949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.958866] LustreError: 130948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.708849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4079.076367] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4084.014345] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4092.379877] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4098.853673] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4106.767922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.240852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.243768] Lustre: Skipped 1 previous similar message [ 4113.780750] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4121.974248] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4130.163394] Lustre: *** cfs_fail_loc=159, val=0*** [ 4146.436312] Lustre: lustre-MDT0000: Client 92328f3c-4289-4c6d-ad79-6611f8b3ef15 (at 192.168.202.52@tcp) reconnecting [ 4147.935401] Lustre: *** cfs_fail_loc=159, val=0*** [ 4147.939675] Lustre: Skipped 5 previous similar messages [ 4164.314224] Lustre: lustre-MDT0000: Client 92328f3c-4289-4c6d-ad79-6611f8b3ef15 (at 192.168.202.52@tcp) reconnecting [ 4184.038891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.051781] Lustre: Skipped 1 previous similar message [ 4184.054071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4184.060884] Lustre: Skipped 1 previous similar message [ 4188.179545] Lustre: server umount lustre-OST0000 complete [ 4194.280358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.300558] Lustre: Skipped 1 previous similar message [ 4198.261890] Lustre: server umount lustre-MDT0000 complete [ 4199.401887] LustreError: 133559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4199.417770] LustreError: 133559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4202.400987] LustreError: 133680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789086724 with bad export cookie 12264536830036448400 [ 4202.408369] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.776884] Lustre: server umount lustre-MDT0001 complete [ 4215.882883] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4218.841554] Key type lgssc unregistered [ 4219.178867] LNet: 134739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4219.185840] LNetError: 134739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4219.206854] LNet: Removed LNI 192.168.202.152@tcp [ 4220.109939] Key type .llcrypt unregistered [ 4220.114113] Key type ._llcrypt unregistered [ 4247.152509] Key type ._llcrypt registered [ 4247.154619] Key type .llcrypt registered [ 4247.247087] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4248.262919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4248.333602] alg: No test for adler32 (adler32-zlib) [ 4249.360540] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 4249.714479] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4251.400968] Key type lgssc registered [ 4252.612209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4261.625357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4261.681771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4263.147737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.828305] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4273.633508] LustreError: 136165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.644832] LustreError: 136165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4278.388939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.945396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4284.849261] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4294.783858] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4301.764969] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4310.441084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4311.071133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4318.577081] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4328.230976] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4333.615303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4344.763857] Lustre: *** cfs_fail_loc=159, val=0*** [ 4361.002819] Lustre: lustre-MDT0000: Client 72c4aec5-c920-4e92-8520-414d8552e813 (at 192.168.202.52@tcp) reconnecting [ 4364.678818] Lustre: *** cfs_fail_loc=159, val=0*** [ 4380.957586] Lustre: lustre-MDT0001: Client 72c4aec5-c920-4e92-8520-414d8552e813 (at 192.168.202.52@tcp) reconnecting [ 4383.030742] Lustre: *** cfs_fail_loc=159, val=0*** [ 4383.046654] Lustre: Skipped 3 previous similar messages [ 4399.401983] Lustre: lustre-MDT0001: Client 72c4aec5-c920-4e92-8520-414d8552e813 (at 192.168.202.52@tcp) reconnecting [ 4415.359634] Lustre: 139257:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4418.055522] Lustre: *** cfs_fail_loc=159, val=0*** [ 4418.075368] Lustre: Skipped 4 previous similar messages [ 4433.712191] Lustre: lustre-MDT0000: Client 1329497e-fcb7-448b-9741-a10fa6243b7e (at 192.168.202.52@tcp) reconnecting [ 4435.347205] Lustre: *** cfs_fail_loc=159, val=0*** [ 4435.349105] Lustre: Skipped 14 previous similar messages [ 4450.585292] Lustre: lustre-MDT0000: Client 1329497e-fcb7-448b-9741-a10fa6243b7e (at 192.168.202.52@tcp) reconnecting [ 4453.193332] Lustre: 139548:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4471.783985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4471.786922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.794774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4471.811070] Lustre: Skipped 1 previous similar message [ 4476.128578] Lustre: server umount lustre-OST0000 complete [ 4482.017286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.018289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.052021] Lustre: Skipped 1 previous similar message [ 4482.078744] Lustre: Skipped 2 previous similar messages [ 4486.985073] Lustre: server umount lustre-MDT0000 complete [ 4487.136270] LustreError: 139376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4487.169715] LustreError: 139376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4491.011357] LustreError: 136145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087013 with bad export cookie 5394725001407826818 [ 4491.014138] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4491.029150] LustreError: 136145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4491.651523] Lustre: server umount lustre-MDT0001 complete [ 4503.753196] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4506.740631] Key type lgssc unregistered [ 4507.058801] LNet: 140627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4507.066336] LNetError: 140627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4507.095619] LNet: Removed LNI 192.168.202.152@tcp [ 4507.922799] Key type .llcrypt unregistered [ 4507.924516] Key type ._llcrypt unregistered [ 4531.283242] Key type ._llcrypt registered [ 4531.285759] Key type .llcrypt registered [ 4531.396237] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4532.532362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4532.584702] alg: No test for adler32 (adler32-zlib) [ 4533.627764] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 4533.835326] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4535.503761] Key type lgssc registered [ 4536.744399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4544.294913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4544.311077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.703561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.703576] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4555.251297] LustreError: 142036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.262615] LustreError: 142036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4557.722121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.062672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4561.742568] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4568.240690] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4575.221499] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4583.772586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.514122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4590.648429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4591.070861] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4597.417505] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4603.968718] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4606.306562] Lustre: 144616:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4608.649669] Lustre: 144664:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4609.991937] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4610.886651] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4614.073281] Lustre: 144812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4614.823123] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4615.737870] Lustre: 144910:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4621.280920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4621.294064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.344722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4624.363425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4624.365706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4624.366489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4625.803360] Lustre: server umount lustre-OST0000 complete [ 4629.985650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4629.990050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4630.002837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4630.013907] Lustre: Skipped 1 previous similar message [ 4634.594961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4634.601243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.715737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4639.721559] Lustre: Skipped 2 previous similar messages [ 4644.320264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4644.570142] Lustre: server umount lustre-MDT0000 complete [ 4644.835498] LustreError: 142036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4644.853723] LustreError: 142036:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4648.621610] LustreError: 142017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087170 with bad export cookie 11977808724789128288 [ 4648.624038] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.631955] LustreError: 142017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4649.221913] Lustre: server umount lustre-MDT0001 complete [ 4660.923444] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4663.827643] Key type lgssc unregistered [ 4664.166876] LNet: 145969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4664.180374] LNetError: 145969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4664.199789] LNet: Removed LNI 192.168.202.152@tcp [ 4665.041935] Key type .llcrypt unregistered [ 4665.045839] Key type ._llcrypt unregistered [ 4687.947470] Key type ._llcrypt registered [ 4687.949696] Key type .llcrypt registered [ 4688.093572] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4689.957026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4690.025676] alg: No test for adler32 (adler32-zlib) [ 4691.194265] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 4691.451388] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4693.143165] Key type lgssc registered [ 4694.567598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4704.750076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4704.764535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.384088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4711.966567] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4716.515976] LustreError: 147387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4716.539530] LustreError: 147387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4721.635994] LustreError: 147388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.254907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4723.000552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4728.958966] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4736.454166] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4742.240345] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4750.074295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.532848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.602498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4758.722924] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4768.006047] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4768.149168] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4776.164400] Lustre: *** cfs_fail_loc=159, val=0*** [ 4791.591679] Lustre: lustre-MDT0000: Client 5583156e-aaf8-41b2-afd1-2e6aac3cc01c (at 192.168.202.52@tcp) reconnecting [ 4809.194995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4809.209893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4809.230083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4811.409026] Lustre: server umount lustre-OST0000 complete [ 4819.434580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4819.440131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4819.462412] Lustre: Skipped 2 previous similar messages [ 4819.493860] Lustre: Skipped 2 previous similar messages [ 4821.850403] Lustre: server umount lustre-MDT0000 complete [ 4824.552217] LustreError: 147387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4824.575199] LustreError: 147387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4825.352097] LustreError: 147369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087347 with bad export cookie 9816886162565883021 [ 4825.360244] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4825.714131] Lustre: server umount lustre-MDT0001 complete [ 4839.885399] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 4843.694238] Key type lgssc unregistered [ 4844.082698] LNet: 151058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4844.092506] LNetError: 151058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4844.116925] LNet: Removed LNI 192.168.202.152@tcp [ 4845.054983] Key type .llcrypt unregistered [ 4845.056899] Key type ._llcrypt unregistered [ 4872.413554] Key type ._llcrypt registered [ 4872.418409] Key type .llcrypt registered [ 4872.513955] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 4874.007234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4874.038086] alg: No test for adler32 (adler32-zlib) [ 4875.038289] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 4875.241090] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 4876.927224] Key type lgssc registered [ 4878.365217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4886.487766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4886.506140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.168176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4894.251111] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4898.801375] LustreError: 152475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.811471] LustreError: 152475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4903.627234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.905496] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4904.246205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4909.714796] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4918.617298] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4926.200137] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4934.738411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4935.042969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4942.116372] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 4949.452565] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4950.783761] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4958.667951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4980.596104] Lustre: lustre-OST0000: evicting a34764f1-4bcf-41fa-a634-9f4612b5056c (at 192.168.202.52@tcp) by administrative request [ 4980.596117] Lustre: lustre-MDT0001: evicting a34764f1-4bcf-41fa-a634-9f4612b5056c (at 192.168.202.52@tcp) by administrative request [ 5009.890066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5009.896411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5009.915887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5009.936064] Lustre: Skipped 1 previous similar message [ 5009.962214] Lustre: Skipped 1 previous similar message [ 5015.010763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5015.027442] Lustre: Skipped 1 previous similar message [ 5015.529203] Lustre: server umount lustre-OST0000 complete [ 5020.131294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5020.158319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5020.168477] Lustre: Skipped 1 previous similar message [ 5025.250375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5025.261649] Lustre: Skipped 1 previous similar message [ 5025.826471] Lustre: server umount lustre-MDT0000 complete [ 5029.652260] LustreError: 152454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087551 with bad export cookie 5704370565519680923 [ 5029.673332] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5029.958946] Lustre: server umount lustre-MDT0001 complete [ 5042.001220] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5045.348766] Key type lgssc unregistered [ 5045.639219] LNet: 156566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5045.653053] LNetError: 156566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5045.672258] LNet: Removed LNI 192.168.202.152@tcp [ 5046.744994] Key type .llcrypt unregistered [ 5046.753240] Key type ._llcrypt unregistered [ 5137.403549] Key type ._llcrypt registered [ 5137.408532] Key type .llcrypt registered [ 5137.490537] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5138.624611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5138.959711] alg: No test for adler32 (adler32-zlib) [ 5140.127745] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 5140.356032] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5142.047174] Key type lgssc registered [ 5143.300564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5151.220512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5151.253825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5152.788662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5157.694344] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5162.977621] LustreError: 159059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.995643] LustreError: 159059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5166.132933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.348823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5170.934462] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5179.357973] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5185.679253] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5193.245731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5193.772837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5197.875210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5200.491732] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5209.226917] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5211.196069] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5221.354383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.359526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5221.381115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5221.392310] Lustre: Skipped 1 previous similar message [ 5225.265408] Lustre: server umount lustre-OST0000 complete [ 5231.585041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.592030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5231.600400] Lustre: Skipped 2 previous similar messages [ 5235.014236] Lustre: server umount lustre-MDT0000 complete [ 5236.704746] LustreError: 159055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5236.712984] LustreError: 159055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5238.818721] LustreError: 159040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087760 with bad export cookie 13959017314338110640 [ 5238.830462] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5238.834357] LustreError: 159040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5239.386572] Lustre: server umount lustre-MDT0001 complete [ 5250.979028] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5253.999348] Key type lgssc unregistered [ 5254.288406] LNet: 162601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5254.297544] LNetError: 162601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5254.309978] LNet: Removed LNI 192.168.202.152@tcp [ 5255.169828] Key type .llcrypt unregistered [ 5255.172126] Key type ._llcrypt unregistered [ 5278.716957] Key type ._llcrypt registered [ 5278.718811] Key type .llcrypt registered [ 5278.842266] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5280.527673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5280.561891] alg: No test for adler32 (adler32-zlib) [ 5281.658972] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 5281.953284] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5283.639271] Key type lgssc registered [ 5284.901126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5302.583678] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 20:50:21 (1789087821) [ 5319.153153] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 5329.941788] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5370.136549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5370.172154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.381524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5371.400459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5371.450156] Lustre: lustre-MDT0000: new disk, initializing [ 5371.506225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.523629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5376.468968] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5388.542609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.758175] Lustre: 166872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5388.868147] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5388.872095] Lustre: Skipped 1 previous similar message [ 5389.000868] Lustre: lustre-MDT0001: new disk, initializing [ 5389.087119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5389.135628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5389.152982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5394.642312] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5404.324620] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5410.853647] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5418.627412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5419.084759] Lustre: lustre-OST0000: new disk, initializing [ 5419.089325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5419.102780] Lustre: 168840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5419.221831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5425.825176] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5428.279281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5428.295778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5428.359943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5436.881596] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5449.184641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5449.187467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5449.189031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5449.213686] Lustre: Skipped 1 previous similar message [ 5452.789477] Lustre: server umount lustre-OST0000 complete [ 5459.425591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.430562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5459.436908] Lustre: Skipped 1 previous similar message [ 5459.450019] Lustre: Skipped 2 previous similar messages [ 5463.613848] Lustre: server umount lustre-MDT0000 complete [ 5468.072156] LustreError: 166862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789087990 with bad export cookie 9577078731278912569 [ 5468.075799] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5468.086649] LustreError: 166862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5468.592320] Lustre: server umount lustre-MDT0001 complete [ 5479.639023] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 5482.669202] Key type lgssc unregistered [ 5482.997668] LNet: 170743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5483.002282] LNetError: 170743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5483.027036] LNet: Removed LNI 192.168.202.152@tcp [ 5484.076657] Key type .llcrypt unregistered [ 5484.084085] Key type ._llcrypt unregistered [ 5512.028875] Key type ._llcrypt registered [ 5512.030843] Key type .llcrypt registered [ 5512.168232] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 5513.266889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5513.445938] alg: No test for adler32 (adler32-zlib) [ 5514.549414] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 5514.829894] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 5516.535361] Key type lgssc registered [ 5517.834800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5526.205785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5526.237386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.821949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5532.973932] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5537.252151] LustreError: 172159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.275149] LustreError: 172159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5541.494853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.751084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5546.466552] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5555.756072] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5561.974150] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5568.923931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5569.397150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5572.481028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5575.532636] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 5584.040087] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5593.230839] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5593.261848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.291320] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5593.305657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.325571] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5594.343827] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5594.375655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.390537] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5594.407606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.423494] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5595.453505] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5595.485406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.498222] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5595.501738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.511408] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5596.707157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.745641] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5597.753019] Lustre: Skipped 1 previous similar message [ 5597.779752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.787395] Lustre: Skipped 1 previous similar message [ 5597.791802] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5597.795173] Lustre: Skipped 1 previous similar message [ 5597.799854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.820700] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5597.823184] Lustre: Skipped 1 previous similar message [ 5598.807397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.955039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.263623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.364218] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5602.379868] Lustre: Skipped 3 previous similar messages [ 5602.413855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.424251] Lustre: Skipped 3 previous similar messages [ 5602.432635] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5602.435766] Lustre: Skipped 3 previous similar messages [ 5602.438139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.443622] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5602.446802] Lustre: Skipped 3 previous similar messages [ 5603.483234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.575419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.417166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.338350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.348815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.353899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.338836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.226056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.192875] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5611.196800] Lustre: Skipped 8 previous similar messages [ 5611.212916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5611.225516] Lustre: Skipped 8 previous similar messages [ 5611.229746] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5611.235254] Lustre: Skipped 8 previous similar messages [ 5611.238755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.246734] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5611.252962] Lustre: Skipped 8 previous similar messages [ 5612.144387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.208299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.449406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.402234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.412655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.388670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.428073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.369620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.563226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.559541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.536091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.450716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.567373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.703055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.610297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.600639] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5627.607385] Lustre: Skipped 15 previous similar messages [ 5627.633398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.642926] Lustre: Skipped 15 previous similar messages [ 5627.647039] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5627.651698] Lustre: Skipped 15 previous similar messages [ 5627.655195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.663121] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5627.666085] Lustre: Skipped 15 previous similar messages [ 5628.801312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.760176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.835438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.876561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.691846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.728766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.758700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.577720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.611765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.510611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.512069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.512911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.629874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.475599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.711671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.615043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.647284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.340626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.107759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.994838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.909475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.710558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.728844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.708249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.489887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.129362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.741525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.581058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.542359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.400223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.538334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.938833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.190809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.747660] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5659.751675] Lustre: Skipped 33 previous similar messages [ 5659.762456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.773217] Lustre: Skipped 33 previous similar messages [ 5659.777492] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5659.784207] Lustre: Skipped 33 previous similar messages [ 5659.787603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.794655] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5659.797068] Lustre: Skipped 33 previous similar messages [ 5660.898562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.567989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.188292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.262258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.738697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.844405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.000257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.137359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.700238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.425448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.453723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.459497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.388331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.449082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.573314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.656937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.743104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.850353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.847914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.665826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.637116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.415504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.079426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.057340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.104984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.079134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.956061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.980272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.638737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.558779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.150727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.992868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.899459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.627724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.564407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.495051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.376135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.379349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.362695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.332374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.567713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.492504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.700324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.689617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.080424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.253745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.520552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.525091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.634626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.888866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.009767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.905646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.038445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.554878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.447896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.621695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.954684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.137816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.537489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.106082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.061653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.808466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.726726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.700852] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5724.709882] Lustre: Skipped 63 previous similar messages [ 5724.733532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.744865] Lustre: Skipped 63 previous similar messages [ 5724.748987] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5724.751196] Lustre: Skipped 63 previous similar messages [ 5724.753982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.761955] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5724.764897] Lustre: Skipped 63 previous similar messages [ 5725.546854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.602859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.643235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.593758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.228805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.177153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.005950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.825451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.807538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.991117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.021500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.181691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.360747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.478599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.617279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.489282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.779707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.791865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.904960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.002476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.174908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.453230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.597658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.412769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.330470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.165461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.012039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.784228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.394646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.967346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.882349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.683903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.866859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.844775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.014256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.815239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.912208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.759626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.766581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.829811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.918402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.918609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.984707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.002478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.309726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.617484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.698233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.922550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.822680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.842480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.738566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.879216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.610856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.613604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.590285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.497272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.408620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.572141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.805809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.813874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.992196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.976963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.052426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.151520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.181877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.218290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.071612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.918854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.848205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.874484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.704758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.652708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.373008] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.324253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.179484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.007337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.696798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.831536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.800968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.880145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.015518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.037214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.403163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.476559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.355837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.296170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.157056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.374621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.703526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.185471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.219452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.152091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.175379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.443795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.811740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.706167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.686722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.546043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.699330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.685636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.816867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.773156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.874197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.959711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.899476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.949287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.906550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.034239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.099712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.002235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.052917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.981375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.073849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.956279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.853802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.792625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.819107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.775358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.727120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.523217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.630065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.657906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.714348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.720025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.667981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.730992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.449588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.404574] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5853.410493] Lustre: Skipped 127 previous similar messages [ 5853.438669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5853.453248] Lustre: Skipped 127 previous similar messages [ 5853.459925] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5853.465530] Lustre: Skipped 127 previous similar messages [ 5853.469820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.491789] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5853.494738] Lustre: Skipped 127 previous similar messages [ 5854.570257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.658645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.693565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.715525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.647692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.640550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.630503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.504950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.527468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.591322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.549917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.668114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.676769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.605707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.436865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.370036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.525872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.494798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.417156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.601357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.568984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.775517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.830496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.896686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.926761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.951281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.989432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.807376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.872812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.919470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.843959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.959848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.077645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.036291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.999592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.957594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.825226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.804591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.633716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.670814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.555163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.502856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.421133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.381820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.212797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.149620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.216196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.570856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.376845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.267160] LustreError: 172180:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5903.278464] LustreError: 172180:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5903.282789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.348863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.655461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.683234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.740489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.565731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.525961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.507559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.560879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.531574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.631362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.516707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.307274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.118428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.084217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.036474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.981220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.856867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.824114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.705571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.540535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.528040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.380585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.150288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.966981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.793065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.820307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.657858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.625638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.375906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.499577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.607713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.644594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.717608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.721939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.497926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.319657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.109668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.029181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.830579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.827502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.754957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.542593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.392430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.203331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.029487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.864323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.713040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.606764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.406260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.293773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.028140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.871647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.752993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.444860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.231664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.088875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.862803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.666240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.548771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.545898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.254483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.016236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.934406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.016621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.876700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.863430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.680268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.580500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.275199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.104824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.803375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.695457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.401580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.182554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.109135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.135426] LustreError: 172180:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5971.149403] LustreError: 172180:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5971.181934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.325333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.267710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.312794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.277716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.577027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.516159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.569643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.759469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.708690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.050419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.165827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.224641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.399625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.709734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.973439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.918956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.030303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.182604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.230935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.383550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.163670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.974868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.010597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.273749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.216419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.122541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.979693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.755921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.507426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.588105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.542492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.486366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.388799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.313512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.235938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.212443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.984995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.722435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.534160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.513764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.517565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.321177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.458904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.330086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.302234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.321132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.171729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.908978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.783976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.631994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.559539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.530759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.488356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.428722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.565797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.461844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.520309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.518352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.343329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.130394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.951984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.690389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.579686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.707339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.711922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.789765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.764137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.870340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.064876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.171230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.134560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.138679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.324601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.303916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.247819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.414878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.649146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.952505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.953931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.216051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.271130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.425681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.552671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.227736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.076354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.921273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.053143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.861800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.790332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.734510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.520605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.427531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.216683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.082506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.947810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.799333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.589984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.492958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.209212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.117573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.033788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.102622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.914477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.049301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.988538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.060669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.968978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.907218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.022404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.052191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.934375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.956740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.723919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.685270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.596485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.516380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.399434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.330580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.261720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.255200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.097973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.988486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.926985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.975460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.727447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.637307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.551252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.449170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.394953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.482208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.448740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.557255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.440576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.350157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.349273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.495029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.594795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.635769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.586424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.701346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.562127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.388917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.157330] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6110.161700] Lustre: Skipped 268 previous similar messages [ 6110.174525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.188031] Lustre: Skipped 268 previous similar messages [ 6110.192909] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6110.198173] Lustre: Skipped 268 previous similar messages [ 6110.201368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.220660] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6110.227639] Lustre: Skipped 268 previous similar messages [ 6110.950611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.701272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.530336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.323810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.281470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.122340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.047556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.886247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.975625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.942745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.855039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.650968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.728809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.464990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.269093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.177621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.087576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.885347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.646533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.886469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.901450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.009598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.957429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.963278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.163263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.116736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.098410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.049615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.035461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.909053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.725655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.574234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.518599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.606717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.540303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.369356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.192433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.009945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.793880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.693784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.325251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.259402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.052504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.778433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.400032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.234051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.918615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.751564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.599175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.522173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.214468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.093981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.967805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.793677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.596170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.521427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.413609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.416888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.087094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.041588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.083500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.233137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.133402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.117702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.114491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.041710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.046606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.058789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.101346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.095352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.097779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.314435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.358795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.402532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.228405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.074231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.961698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.729161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.639915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.323176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.092052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.889047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.800870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.464673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.253068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.996665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.883923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.690434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.405321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.214733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.178284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.986290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.028509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.967967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.924514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.007189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.928419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.780556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.530934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.668886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.001732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.240086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.224403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.230154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.385292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.463854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.741905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.959097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.480646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.697952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.619906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.792697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.946991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.972294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.482723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.446049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.793404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.026525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.027313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.149085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.459052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.427765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.593490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.610691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.115603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.187137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.451808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.386375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.537796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.689908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.398889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.061149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.052830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.026768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.283314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.470590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.541858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.536217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.672880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.704750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.700085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.836457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.806753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.631128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.627941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.841764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.830848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.858871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.885287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.994831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.196241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.150924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.258358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.223211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.317102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.378677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.548076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.842230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.085253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.992262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.902689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.919348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.850564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.817618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.793573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.566147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.427526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.210294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.036726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.905445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.908376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.805875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.838286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.660566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.695631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.851993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.017313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.939686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.009276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.935365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.157841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.383779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.437419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.292853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.385725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.491978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.220094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.059086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.953759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.636501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.597298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.514152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.378385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.334824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.229876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.068229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.112759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.070890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.233588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.225585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.201522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.991363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.119366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.922878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.736484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.441226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.300530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.078911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.182268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.267251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.161444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.936324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.923185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.147602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.043049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.201618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.357913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.233656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.813900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.316789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.898280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.642607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.579214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.579682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.970878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.962691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.072221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.961640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.329540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.695498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.025685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.234175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.319202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.451963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.496064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.445397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.435188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.307044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.966305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.599927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.195291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.989386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.694159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.476530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.675327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.390705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.244052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.227457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.217438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.320082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.196787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.259304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.884281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.509803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.329606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.516269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.441107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.253728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.189534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.052931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.907798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.670376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.587978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.508540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.494195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.392220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.449928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.435575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.378648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.266045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.228700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.102207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.884673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.647323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.634848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.466194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.347786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.220173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.180393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.134068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.103181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.101531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.855833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.626417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.388268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.266763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.102132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.029779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.850825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.447686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.250588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.313707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.197721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.872351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.978856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.703752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.535897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.313711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.470731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.485083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.233091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.143596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.191612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.288969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.421214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.585796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.772333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.989597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.878995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.144214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.202722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.167779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.059407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.031661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.016581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.192086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.991323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.091846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.039448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.287373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.312826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.435846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.437310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.694809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.282816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.445778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.498383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.410400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.268427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.671054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.849646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.930352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.033910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.203725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.171351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.050359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.363323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.160378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.341897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.279916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.675650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.695292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.870683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.769255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.684745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.640952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.625824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.567771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.500441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.373555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.506202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.191842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.631842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.613504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.620509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.545300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.480402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.270429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.042252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.778840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.635821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.782037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.750834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.872441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.883349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.846282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.796972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.828880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.977243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.164642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.144613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.498165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.702853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.457156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.070846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.897886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.648619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.653074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.402897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.487536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.254600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.415461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.499502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.573201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.613651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.740565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.708100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.710148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.696778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.813453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.814852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.940651] LustreError: 172180:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6493.961453] LustreError: 172180:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6493.968982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.768266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.859663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.857301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.937713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.851057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.898474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.683988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.444905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.234103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.018961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.773204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.642900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.342654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.165319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.016503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.838061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.505802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.456144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.241248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.270467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.068764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.979716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.862400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.703985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.369752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.215500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.115097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.062304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.931433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.764322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.564765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.449185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.322857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.147126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.837992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.682603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.387935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.180147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.989309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.834822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.702883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.611572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.369303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.171689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.963796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.728507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.479576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.133717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.809608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.652356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.406773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.239508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.002199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.784400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.577965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.379145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.183805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.109814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.968305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6543.799972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.615093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.551497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.479906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.254463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.023555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.100967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.073051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.098823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6552.127979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.274386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.223065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.057255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.278191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.269680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.290843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.402355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.465672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.586829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.489498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6944.448980] LustreError: 174651:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6944.455954] LustreError: 172180:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7637.992815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.006375] Lustre: Skipped 471 previous similar messages [ 7638.017244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7638.024443] Lustre: Skipped 1 previous similar message [ 7643.124021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7643.132323] Lustre: Skipped 1 previous similar message [ 7643.471052] Lustre: server umount lustre-OST0000 complete [ 7648.227912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7653.344625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7653.359218] Lustre: Skipped 2 previous similar messages [ 7654.629955] Lustre: server umount lustre-MDT0000 complete [ 7658.466176] LustreError: 172159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.483161] LustreError: 172159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7658.798749] LustreError: 172141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090180 with bad export cookie 16694812501370165077 [ 7658.836331] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.326070] Lustre: server umount lustre-MDT0001 complete [ 7671.714220] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7674.160507] Key type lgssc unregistered [ 7674.468227] LNet: 227427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7674.485691] LNetError: 227427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7674.513547] LNet: Removed LNI 192.168.202.152@tcp [ 7675.420966] Key type .llcrypt unregistered [ 7675.425765] Key type ._llcrypt unregistered [ 7704.282676] Key type ._llcrypt registered [ 7704.285278] Key type .llcrypt registered [ 7704.386938] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7705.426244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7705.555839] alg: No test for adler32 (adler32-zlib) [ 7706.671323] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 7706.883380] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7708.535178] Key type lgssc registered [ 7709.895995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7717.408452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7717.452085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7719.052956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.248911] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7729.633838] LustreError: 228845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7729.645754] LustreError: 228845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7730.661073] LustreError: 228844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7733.381444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7733.686140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7738.444317] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7747.003376] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7753.033594] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7760.352153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7760.760645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.894635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7767.688527] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7776.183257] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7785.491942] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7787.377953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7787.385082] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7787.392589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7787.411794] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7787.414688] LustreError: 228864:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7794.439640] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 7794.657992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7794.666701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.689346] Lustre: Skipped 1 previous similar message [ 7799.524853] Lustre: lustre-OST0000: Not available for connect from 192.168.202.52@tcp (stopping) [ 7799.531241] Lustre: Skipped 2 previous similar messages [ 7804.384149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7804.625135] LustreError: 230957:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.720132] Lustre: server umount lustre-OST0000 complete [ 7807.455336] LustreError: 228864:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7839.445591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.947319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.808491] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7849.128940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7849.139259] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7849.147671] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7849.157057] Lustre: Skipped 1 previous similar message [ 7853.930933] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7862.521671] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 7870.502460] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8951475c6000.ost_server_uuid 50 [ 7871.993889] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8951475c6000.ost_server_uuid in FULL state after 0 sec [ 7877.089175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7877.093172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.125735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7877.132483] Lustre: Skipped 2 previous similar messages [ 7878.845919] Lustre: server umount lustre-OST0000 complete [ 7885.290330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7885.295653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7885.314743] Lustre: Skipped 1 previous similar message [ 7885.335835] Lustre: Skipped 1 previous similar message [ 7889.864523] Lustre: server umount lustre-MDT0000 complete [ 7890.401450] LustreError: 228839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.420396] LustreError: 228839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7893.961977] LustreError: 228825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090415 with bad export cookie 8806161909749020191 [ 7893.964493] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7893.974139] LustreError: 228825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7904.440851] Lustre: server umount lustre-MDT0001 complete [ 7916.612780] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 7919.405596] Key type lgssc unregistered [ 7919.791575] LNet: 234054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7919.800372] LNetError: 234054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7919.812987] LNet: Removed LNI 192.168.202.152@tcp [ 7920.742954] Key type .llcrypt unregistered [ 7920.747363] Key type ._llcrypt unregistered [ 7958.662916] Key type ._llcrypt registered [ 7958.668588] Key type .llcrypt registered [ 7958.912299] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 7960.414031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7960.458766] alg: No test for adler32 (adler32-zlib) [ 7962.454583] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 7962.844558] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 7964.711437] Key type lgssc registered [ 7966.176341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7974.579947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7974.622989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.273892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7981.783910] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7986.659479] LustreError: 235787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7986.673307] LustreError: 235787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7991.790335] LustreError: 235786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7992.656720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7993.176688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7997.359683] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 7999.866843] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8005.449312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8005.767335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8010.989961] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8019.415088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8019.620431] Lustre: lustre-OST0001: new disk, initializing [ 8019.622499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8019.626482] Lustre: 238092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8019.760849] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8021.833559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8026.382972] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8026.820441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8026.834914] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8026.934835] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8038.307707] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8045.366287] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8059.872410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8059.878915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.895247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8062.945282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8062.951642] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8062.976354] Lustre: Skipped 2 previous similar messages [ 8064.701522] Lustre: server umount lustre-MDT0000 complete [ 8068.069864] LustreError: 235787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8068.081594] LustreError: 235787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8068.355546] LustreError: 235766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090590 with bad export cookie 13469074779888889911 [ 8068.358811] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8068.367971] LustreError: 235766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8068.726614] Lustre: server umount lustre-MDT0001 complete [ 8082.535838] Lustre: server umount lustre-OST0000 complete [ 8096.591825] Lustre: server umount lustre-OST0001 complete [ 8120.913490] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8132.442668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.036671] LustreError: 241201:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8133.141447] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8138.209112] LustreError: 241202:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8138.447244] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8148.462759] LustreError: 241201:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8148.513863] LustreError: 241201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8150.734694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8151.343315] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8157.060247] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8160.167168] Lustre: 242346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8167.694947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8168.245915] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.798515] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8176.441725] LustreError: 242698:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8178.487943] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49760) [ 8183.956038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.035124] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8200.452724] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8204.439845] Lustre: 244217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8218.080626] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8218.087693] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8218.110862] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8218.125821] Lustre: Skipped 3 previous similar messages [ 8220.143058] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8220.143863] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8220.167040] Lustre: Skipped 2 previous similar messages [ 8224.079148] Lustre: server umount mylustre-MDT0000 complete [ 8225.279124] LustreError: 241202:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.307068] LustreError: 241202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 8228.156705] LustreError: 241176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090750 with bad export cookie 13469074779888892704 [ 8228.159336] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8228.167469] LustreError: 241176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8228.603724] Lustre: server umount mylustre-MDT0001 complete [ 8243.377927] Lustre: server umount mylustre-OST0000 complete [ 8246.559808] Lustre: 235048:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789090752/real 1789090752] req@ffff99c5957cdf80 x1875997343507072/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789090768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8246.577788] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.202592] Lustre: server umount mylustre-OST0001 complete [ 8272.893842] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8285.424888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.205056] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8286.216875] Lustre: Skipped 1 previous similar message [ 8291.302294] LustreError: 246528:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8291.320140] LustreError: 246528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8291.807146] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8301.436156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8306.314322] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8310.461815] Lustre: 247669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8321.052510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8329.307862] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8331.871607] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49792) [ 8339.506227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8345.120126] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8346.895389] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8355.730541] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8359.870430] Lustre: 249542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8368.609258] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8368.627346] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8368.644659] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8368.649695] Lustre: Skipped 3 previous similar messages [ 8369.923723] Lustre: server umount tfs-MDT0000 complete [ 8373.577667] LustreError: 246505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789090895 with bad export cookie 13469074779888894370 [ 8373.579374] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8373.595067] LustreError: 246505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8374.254734] Lustre: server umount tfs-MDT0001 complete [ 8388.755163] Lustre: server umount tfs-OST0000 complete [ 8391.711126] Lustre: 235047:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789090897/real 1789090897] req@ffff99c69d051f80 x1875997343560448/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789090913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8391.748492] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8391.762903] Lustre: Skipped 3 previous similar messages [ 8392.959321] Lustre: server umount tfs-OST0001 complete [ 8417.086315] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8427.588781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8427.935267] LustreError: 251794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8427.958752] LustreError: 251794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8428.051399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8428.062553] Lustre: Skipped 3 previous similar messages [ 8432.634951] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8441.304115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8446.705869] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8449.793185] Lustre: 252937:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8456.243129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8463.538586] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8471.899586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8473.130452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49824) [ 8473.160034] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8479.161245] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8488.515512] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8492.249237] Lustre: 254809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8501.237888] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 21:43:39 (1789091019) [ 8519.144557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.149187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8519.169423] Lustre: Skipped 2 previous similar messages [ 8519.183025] Lustre: Skipped 3 previous similar messages [ 8525.761997] Lustre: server umount lustre-MDT0000 complete [ 8532.264190] LustreError: 253314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091054 with bad export cookie 13469074779888896036 [ 8532.289489] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8532.296707] LustreError: 253314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8537.517524] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8537.533404] Lustre: Skipped 7 previous similar messages [ 8539.183645] Lustre: server umount lustre-MDT0001 complete [ 8550.236992] Lustre: server umount lustre-OST0000 complete [ 8561.092784] Lustre: server umount lustre-OST0001 complete [ 8569.520800] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8581.217300] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8624.307865] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8624.307865] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8624.307865] [ 8624.337669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8624.554748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8624.592133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8624.728853] Lustre: lustre-MDT0000: new disk, initializing [ 8624.930785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8624.938059] Lustre: Skipped 3 previous similar messages [ 8624.958029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8630.598909] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8643.025988] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8643.025988] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8643.025988] [ 8643.075598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8643.375324] Lustre: 257995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8643.504809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8643.514475] Lustre: Skipped 1 previous similar message [ 8643.682702] Lustre: lustre-MDT0001: new disk, initializing [ 8643.895656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8643.903522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8648.500177] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8661.063351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8661.532643] Lustre: lustre-OST0000: new disk, initializing [ 8661.538734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8661.545782] Lustre: 259577:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8662.980929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8662.986535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8663.115419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8667.217374] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8677.619821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8677.827880] Lustre: lustre-OST0001: new disk, initializing [ 8677.843146] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8677.871473] Lustre: 260451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8679.847297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8679.860822] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8680.042936] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8686.720910] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8700.102827] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 21:46:58 (1789091218) [ 8718.155823] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 21:47:16 (1789091236) [ 8734.689449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8734.707666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8734.730849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8734.751107] Lustre: Skipped 4 previous similar messages [ 8739.810791] Lustre: server umount lustre-OST0000 complete [ 8741.366207] LustreError: 259560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8741.376759] LustreError: 259560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 8746.492227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8746.503140] LustreError: Skipped 1 previous similar message [ 8750.720588] Lustre: server umount lustre-MDT0000 complete [ 8754.930194] LustreError: 257987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091276 with bad export cookie 13469074779888897583 [ 8754.933290] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8754.952792] LustreError: 257987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8761.460994] Lustre: server umount lustre-MDT0001 complete [ 8774.355415] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8788.954632] Lustre: server umount lustre-OST0001 complete [ 8796.237459] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 8813.598950] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8855.247235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.550323] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8855.595236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8855.714203] Lustre: lustre-MDT0000: new disk, initializing [ 8855.889948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8860.415883] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8871.304726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8871.488153] Lustre: 265030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8871.932573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8871.949598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8876.107776] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8887.396635] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8895.284471] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8903.490918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8903.712595] Lustre: lustre-OST0000: new disk, initializing [ 8903.715034] Lustre: Skipped 1 previous similar message [ 8903.717784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8903.720909] Lustre: Skipped 2 previous similar messages [ 8903.726516] Lustre: 267000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8903.804232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8903.809805] Lustre: Skipped 5 previous similar messages [ 8905.859136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8905.885333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8906.040035] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8910.570862] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 8922.093768] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8931.808931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8931.811533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8931.823670] LustreError: Skipped 1 previous similar message [ 8931.828879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8931.852784] Lustre: Skipped 5 previous similar messages [ 8931.855170] Lustre: Skipped 8 previous similar messages [ 8936.617064] Lustre: server umount lustre-OST0000 complete [ 8947.452178] Lustre: server umount lustre-MDT0000 complete [ 8951.381385] LustreError: 265020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091473 with bad export cookie 13469074779888899060 [ 8951.382633] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8951.405037] LustreError: 265020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8951.704979] Lustre: server umount lustre-MDT0001 complete [ 8964.017155] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 8967.170165] Key type lgssc unregistered [ 8967.451818] LNet: 268759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8967.455836] LNetError: 268759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8967.470560] LNet: Removed LNI 192.168.202.152@tcp [ 8968.342372] Key type .llcrypt unregistered [ 8968.352120] Key type ._llcrypt unregistered [ 8997.185993] Key type ._llcrypt registered [ 8997.188203] Key type .llcrypt registered [ 8997.303919] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 8999.322925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8999.365989] alg: No test for adler32 (adler32-zlib) [ 9000.696650] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 9001.116699] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 9002.871329] Key type lgssc registered [ 9004.307577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9011.359937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9011.370500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.757669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9018.249597] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9022.945232] LustreError: 270226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9022.968408] LustreError: 270226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9027.761562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.072154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9028.176087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9032.639466] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9038.678810] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9044.892809] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9052.738209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9053.172534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9057.292091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9060.928978] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9068.890295] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9070.276786] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9072.609691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9072.612606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9072.622285] Lustre: Skipped 1 previous similar message [ 9072.638345] Lustre: Skipped 1 previous similar message [ 9077.741866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9077.749555] Lustre: Skipped 1 previous similar message [ 9078.324127] Lustre: server umount lustre-OST0000 complete [ 9082.849086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9082.852512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9082.858839] Lustre: Skipped 1 previous similar message [ 9082.867706] Lustre: Skipped 1 previous similar message [ 9087.969783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9087.977378] Lustre: Skipped 1 previous similar message [ 9088.248447] Lustre: server umount lustre-MDT0000 complete [ 9091.813137] LustreError: 270207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091613 with bad export cookie 5484666310582623184 [ 9091.819246] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9091.828110] LustreError: 270207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9092.218468] Lustre: server umount lustre-MDT0001 complete [ 9102.936269] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 9105.620899] Key type lgssc unregistered [ 9105.972574] LNet: 273811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9105.979551] LNetError: 273811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9106.001859] LNet: Removed LNI 192.168.202.152@tcp [ 9106.826538] Key type .llcrypt unregistered [ 9106.829923] Key type ._llcrypt unregistered [ 9123.160574] Key type ._llcrypt registered [ 9123.161943] Key type .llcrypt registered [ 9123.248899] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9124.359761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9124.441098] alg: No test for adler32 (adler32-zlib) [ 9125.511555] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 9125.870565] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 9127.519648] Key type lgssc registered [ 9128.568641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9136.986239] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9136.999199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9152.543444] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9157.663417] LustreError: 275009:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.152@tcp: failed processing log, type 4: rc = -110 [ 9184.415219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9191.004928] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9197.979995] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9200.089185] Lustre: server umount lustre-OST0000 complete [ 9216.389469] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 9219.365782] Key type lgssc unregistered [ 9219.728642] LNet: 276640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9219.737444] LNetError: 276640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9219.753278] LNet: Removed LNI 192.168.202.152@tcp [ 9220.557627] Key type .llcrypt unregistered [ 9220.560737] Key type ._llcrypt unregistered [ 9257.317929] Key type ._llcrypt registered [ 9257.322223] Key type .llcrypt registered [ 9257.433341] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9258.477169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9258.538104] alg: No test for adler32 (adler32-zlib) [ 9259.502120] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 9259.679384] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 9261.311415] Key type lgssc registered [ 9262.233355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9272.884217] loop0: detected capacity change from 0 to 204800000 [ 9273.057638] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9273.076823] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9273.367697] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9274.280529] loop0: detected capacity change from 0 to 204800000 [ 9274.461177] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9274.468476] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9274.698434] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9275.479827] loop0: detected capacity change from 0 to 204800000 [ 9275.631596] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9275.647485] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9275.736105] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9276.723548] loop0: detected capacity change from 0 to 204800000 [ 9276.943412] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9277.935181] loop0: detected capacity change from 0 to 204800000 [ 9277.948481] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9281.702710] loop0: detected capacity change from 0 to 204800000 [ 9281.730653] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9285.229985] loop0: detected capacity change from 0 to 204800000 [ 9285.245994] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9289.324947] loop0: detected capacity change from 0 to 204800000 [ 9289.349751] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9292.289392] loop0: detected capacity change from 0 to 204800000 [ 9292.360695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9292.391224] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.454560] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9292.458208] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9292.462348] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9292.467130] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9292.474955] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9292.485061] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9292.493861] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9292.510780] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9292.524934] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9292.538574] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9292.551972] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9292.559927] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9292.566617] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9292.574243] Lustre: 280050:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9297.514501] Lustre: server umount MGS complete [ 9298.218085] loop0: detected capacity change from 0 to 204800000 [ 9298.248114] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.459378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9298.481548] Lustre: lustre-MDT0000: Aborting client recovery [ 9299.042499] loop1: detected capacity change from 0 to 204800000 [ 9299.135869] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.189219] Lustre: 280481:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9299.206760] Lustre: 280481:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9299.219648] Lustre: 280481:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9299.226624] Lustre: 280481:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9299.245618] Lustre: 280481:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9299.265571] Lustre: 280481:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9299.273867] Lustre: 280481:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9299.292776] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9299.442094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9299.467072] Lustre: lustre-MDT0001: Aborting client recovery [ 9300.172798] loop2: detected capacity change from 0 to 204800000 [ 9300.214097] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9300.242876] Lustre: 280561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9300.256508] Lustre: 280561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9300.261707] Lustre: 280561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9300.269781] Lustre: 280561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9300.276806] Lustre: 280561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9300.283658] Lustre: 280561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9300.290693] Lustre: 280561:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9300.368674] Lustre: Found index 0 for lustre-OST0000, updating log [ 9300.386440] Lustre: 280580:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9300.422557] Lustre: lustre-OST0000: Aborting recovery [ 9301.095489] loop3: detected capacity change from 0 to 204800000 [ 9301.204613] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9301.244195] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9301.254386] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9301.264485] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9301.283538] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9301.289693] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9301.295694] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9301.303095] Lustre: 280647:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9301.359516] Lustre: 280653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9301.409505] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9301.411622] Lustre: Skipped 1 previous similar message [ 9301.432827] Lustre: lustre-OST0001: Aborting recovery [ 9306.691459] LustreError: 280425:0:(lod_dev.c:512:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -108 [ 9308.650601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9308.652724] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9308.701470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9308.735974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9309.679427] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9309.704950] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9315.073061] LustreError: 280392:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 284us: rc = -2 [ 9315.595404] LustreError: 280389:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1348us: rc = -2 [ 9315.647925] LustreError: 280389:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 18 previous similar messages [ 9330.144228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9330.146068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9330.162820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9330.194467] Lustre: Skipped 3 previous similar messages [ 9335.189922] Lustre: server umount lustre-MDT0000 complete [ 9335.263822] LustreError: 280395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.278653] LustreError: 280395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9335.906688] LustreError: 280376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789091857 with bad export cookie 7674785143104535622 [ 9335.918432] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9335.919831] LustreError: 280376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9340.385277] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9340.386463] LustreError: 280391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9340.403280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9340.442321] LustreError: 280391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9340.474457] Lustre: Skipped 4 previous similar messages [ 9342.574788] Lustre: server umount lustre-MDT0001 complete [ 9349.708869] Lustre: server umount lustre-OST0000 complete [ 9357.478413] Lustre: server umount lustre-OST0001 complete [ 9370.764436] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 21:58:09 (1789091889) [ 9406.341356] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 9417.489606] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9453.046475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.305608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9453.325170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9453.382527] Lustre: lustre-MDT0000: new disk, initializing [ 9453.413648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9453.430873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9457.289638] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9469.699976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9469.894116] Lustre: 284716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9470.002394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9470.009132] Lustre: Skipped 1 previous similar message [ 9470.180654] Lustre: lustre-MDT0001: new disk, initializing [ 9470.332573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9470.384527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9470.410072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9475.655623] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9484.937196] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9491.333787] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9501.783235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9502.123640] Lustre: lustre-OST0000: new disk, initializing [ 9502.127500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9502.133875] Lustre: 286687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9502.265076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.891184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9507.914673] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9508.067866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9508.917966] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9518.777579] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9523.771583] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9533.068811] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9536.501328] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9546.679783] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9585.120772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9585.138130] Lustre: Skipped 2 previous similar messages [ 9585.146525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9589.739694] Lustre: server umount lustre-OST0000 complete [ 9595.362536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9595.371540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9595.385882] Lustre: Skipped 1 previous similar message [ 9595.400054] Lustre: Skipped 2 previous similar messages [ 9600.026156] Lustre: server umount lustre-MDT0000 complete [ 9600.482752] LustreError: 284723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9600.513465] LustreError: 284723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9603.692512] LustreError: 285512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092125 with bad export cookie 7674785143104538338 [ 9603.694152] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9603.703061] LustreError: 285512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9604.125282] Lustre: server umount lustre-MDT0001 complete [ 9613.472319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9618.594965] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9622.980077] Lustre: server umount MGS complete [ 9634.126074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9634.581935] LustreError: 290472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9634.618321] LustreError: 290485:0:(qmt_pool.c:1854:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9634.680378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9638.895801] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9639.910722] LustreError: 290473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9648.783215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9654.061277] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9664.157244] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9671.504457] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9678.187458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9678.459571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9678.463061] Lustre: Skipped 1 previous similar message [ 9683.723477] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9690.203147] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9720.801029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9720.801051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9720.824757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9720.840286] Lustre: Skipped 1 previous similar message [ 9725.880683] Lustre: server umount lustre-OST0000 complete [ 9731.059501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.076468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9731.098497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9731.104887] LustreError: Skipped 1 previous similar message [ 9731.128087] Lustre: Skipped 2 previous similar messages [ 9736.524592] Lustre: server umount lustre-MDT0000 complete [ 9740.260439] LustreError: 290467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9740.275297] LustreError: 290467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9740.643075] LustreError: 293196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092262 with bad export cookie 7674785143104540263 [ 9740.664654] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9741.289596] Lustre: server umount lustre-MDT0001 complete [ 9751.777428] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [ 9754.381833] Key type lgssc unregistered [ 9754.524803] LNet: 294417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9754.529342] LNetError: 294417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9754.550177] LNet: Removed LNI 192.168.202.152@tcp [ 9755.197425] Key type .llcrypt unregistered [ 9755.199370] Key type ._llcrypt unregistered [ 9797.323323] Key type ._llcrypt registered [ 9797.325559] Key type .llcrypt registered [ 9797.424681] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [ 9815.825747] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [ 9817.372874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9817.504495] alg: No test for adler32 (adler32-zlib) [ 9818.598045] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [ 9818.858304] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [ 9820.759406] Key type lgssc registered [ 9822.002816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9866.451775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9866.493147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9867.759505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9867.862600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9867.981823] Lustre: lustre-MDT0000: new disk, initializing [ 9868.039920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9868.050293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9872.557550] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9883.411781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.545340] Lustre: 298381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9883.613749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9883.616623] Lustre: Skipped 1 previous similar message [ 9883.669520] Lustre: lustre-MDT0001: new disk, initializing [ 9883.751430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9883.800396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9883.812731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9888.930759] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9898.840587] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9904.802879] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9911.876965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9912.129342] Lustre: lustre-OST0000: new disk, initializing [ 9912.132234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9912.137094] Lustre: 300348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9912.244912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9918.477603] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [ 9920.570732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9920.581271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9920.682285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9928.108832] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9935.143872] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9946.835610] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9954.962368] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9997.793357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9997.801206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9997.817168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9997.826116] Lustre: Skipped 1 previous similar message [10001.186894] Lustre: server umount lustre-OST0000 complete [10006.496696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10006.500359] LustreError: Skipped 1 previous similar message [10006.502868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10006.512523] Lustre: Skipped 1 previous similar message [10006.518537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10008.032257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10008.034322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10008.047045] Lustre: Skipped 1 previous similar message [10011.783232] Lustre: server umount lustre-MDT0000 complete [10015.704294] LustreError: 298371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092537 with bad export cookie 11568491052131821829 [10015.727391] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10016.136488] Lustre: server umount lustre-MDT0001 complete [10026.124373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10031.108410] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10036.399613] Lustre: server umount MGS complete [10047.284903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10047.677594] LustreError: 304310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.696991] LustreError: 304310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10047.708834] LustreError: 304323:0:(qmt_pool.c:1854:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10047.811302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10052.499762] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10053.088525] LustreError: 304311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10057.206453] LustreError: 304310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.449640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.829480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10065.179664] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10073.190977] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10078.897603] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10086.165685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10086.476139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10092.205347] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10099.398835] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10126.522431] Lustre: server umount lustre-OST0000 complete [10131.939255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10131.941397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10131.966268] Lustre: Skipped 1 previous similar message [10137.058478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10137.070815] Lustre: Skipped 2 previous similar messages [10140.346824] Lustre: server umount lustre-MDT0000 complete [10142.192664] Lustre: 296604:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789092648/real 1789092648] req@00000000755d63a0 x1875999289646976/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789092664 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10142.194206] LustreError: 304311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10142.215901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10142.248720] LustreError: 304311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10144.272893] LustreError: 307036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789092666 with bad export cookie 11568491052131823656 [10144.276473] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10144.289749] LustreError: 307036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10144.923866] Lustre: server umount lustre-MDT0001 complete [10154.920642] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [10157.430941] Key type lgssc unregistered [10157.700664] LNet: 308256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10157.705663] LNetError: 308256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10157.719276] LNet: Removed LNI 192.168.202.152@tcp [10158.751600] Key type .llcrypt unregistered [10158.755137] Key type ._llcrypt unregistered [10180.330137] Key type ._llcrypt registered [10180.332949] Key type .llcrypt registered [10180.422878] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [10193.452822] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10194.127407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10194.168187] alg: No test for adler32 (adler32-zlib) [10195.197406] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [10195.388487] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [10197.015123] Key type lgssc registered [10197.858750] Lustre: Echo OBD driver; http://www.lustre.org/ [10224.812251] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [10240.090465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10240.107465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10241.386902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10241.415672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10241.491736] Lustre: lustre-MDT0000: new disk, initializing [10241.562103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10241.583978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10245.387983] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10255.724499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10256.067294] Lustre: lustre-OST0000: new disk, initializing [10256.077767] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10256.089510] Lustre: Skipped 1 previous similar message [10256.097711] Lustre: 312305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10256.208959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10262.475893] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [10264.669354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10264.676810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10264.698522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11118.864692] Lustre: server umount lustre-MDT0000 complete [11123.235739] LustreError: 311491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789093645 with bad export cookie 16521670450881893203 [11123.247337] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11133.516475] Lustre: server umount lustre-OST0000 complete [11167.662641] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [11183.659573] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [11228.918424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.450732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11229.588068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11229.766956] Lustre: lustre-MDT0000: new disk, initializing [11229.944299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11229.992483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11235.983611] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11246.566832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11246.831793] Lustre: 316672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11246.926721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11246.929950] Lustre: Skipped 1 previous similar message [11247.036406] Lustre: lustre-MDT0001: new disk, initializing [11247.151104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11247.209053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11247.221568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11251.427775] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11262.009337] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11268.165791] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11276.024793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11276.338440] Lustre: lustre-OST0000: new disk, initializing [11276.345281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11276.355607] Lustre: 318640:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11276.478828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11283.044128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11283.054624] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11283.212149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11283.371749] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11292.010620] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11297.143591] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11297.394990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11302.641423] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11302.903851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11309.031190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11309.046550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11309.053069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11309.078613] Lustre: Skipped 1 previous similar message [11309.095445] Lustre: Skipped 1 previous similar message [11311.499289] Lustre: server umount lustre-OST0000 complete [11318.768367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11318.795504] LustreError: Skipped 1 previous similar message [11318.801189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11318.852387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11322.663524] Lustre: server umount lustre-MDT0000 complete [11327.645200] LustreError: 316663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789093849 with bad export cookie 16521670450884834022 [11327.657948] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11327.664756] LustreError: 316663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11328.366627] Lustre: server umount lustre-MDT0001 complete [11332.398502] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11334.617974] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 22:30:53 (1789093853) [11342.590922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.170806] LustreError: 320955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11343.192767] LustreError: 320955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11343.277845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11348.448387] LustreError: 320956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11349.353654] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11353.569857] LustreError: 320955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11358.689774] LustreError: 320956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11359.145407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11359.709831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11366.066127] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11374.406626] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11380.964734] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11388.242821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11388.626830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11395.199276] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11403.616918] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11411.477190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11411.761324] Lustre: lustre-OST0001: new disk, initializing [11411.772140] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11411.784113] Lustre: 323790:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11411.933373] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11419.863977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11419.882071] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11419.970341] LustreError: 324090:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11421.028953] LustreError: 324090:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11421.061179] LustreError: 324090:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11421.491627] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11422.055554] LustreError: 324095:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11424.171332] LustreError: 324095:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11424.185432] LustreError: 324095:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [11428.260797] LustreError: 324090:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11428.278218] LustreError: 324090:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [11434.847461] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11435.127784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11436.518784] LustreError: 324090:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11436.539705] LustreError: 324090:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11441.344212] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff895142eb5000.ost_server_uuid 50 [11443.215543] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff895142eb5000.ost_server_uuid in FULL state after 0 sec [11450.600986] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff895142eb5000.ost_server_uuid 50 [11452.622677] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff895142eb5000.ost_server_uuid in IDLE state after 0 sec [11453.235481] LustreError: 324095:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11453.250985] LustreError: 324095:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 32 previous similar messages [11455.444452] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11470.817141] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11470.829852] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11470.835861] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11470.841112] Lustre: Skipped 2 previous similar messages [11470.852431] Lustre: Skipped 2 previous similar messages [11474.679341] Lustre: lustre-OST0001: Not available for connect from 192.168.202.52@tcp (stopping) [11474.687803] Lustre: Skipped 3 previous similar messages [11479.754480] Lustre: lustre-OST0001: Not available for connect from 192.168.202.52@tcp (stopping) [11479.770910] Lustre: Skipped 2 previous similar messages [11485.154775] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11485.429776] Lustre: server umount lustre-OST0001 complete [11486.715396] LustreError: 322776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11486.742504] LustreError: 322776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11492.321525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11492.330235] LustreError: Skipped 1 previous similar message [11492.340600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11492.371629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11492.378766] Lustre: Skipped 3 previous similar messages [11496.422165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11496.440327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11496.929981] LustreError: 322776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11496.955350] LustreError: 322776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11498.356800] Lustre: server umount lustre-OST0000 complete [11507.171296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11507.187915] Lustre: Skipped 1 previous similar message [11509.578393] Lustre: server umount lustre-MDT0000 complete [11514.472371] LustreError: 320937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789094036 with bad export cookie 16521670450884834904 [11514.481242] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11514.494582] LustreError: 320937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11515.312482] Lustre: server umount lustre-MDT0001 complete [11527.382708] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [11530.835467] Key type lgssc unregistered [11531.251847] LNet: 326483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11531.256506] LNetError: 326483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11531.269954] LNet: Removed LNI 192.168.202.152@tcp [11532.333271] Key type .llcrypt unregistered [11532.338331] Key type ._llcrypt unregistered [11560.664943] Key type ._llcrypt registered [11560.669338] Key type .llcrypt registered [11560.846641] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [11562.676688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11562.972424] alg: No test for adler32 (adler32-zlib) [11564.231807] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [11564.609559] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [11566.351502] Key type lgssc registered [11567.806557] Lustre: Echo OBD driver; http://www.lustre.org/ [11577.223991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11577.254029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11578.846270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11585.038607] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11588.074028] LustreError: 327907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11588.098051] LustreError: 327907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11593.190429] LustreError: 327908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.866041] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11598.333563] LustreError: 327907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.434820] LustreError: 327908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.542410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11604.057422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11609.336489] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11622.760271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11623.382491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11625.395926] LustreError: 329590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11625.433955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11632.978477] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11633.638807] LustreError: 329590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.686364] LustreError: 329590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11645.030099] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11654.123446] LustreError: 329590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11654.139162] LustreError: 329590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11654.438089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11654.853910] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11660.310051] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11662.646915] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11672.041454] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11679.285272] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11679.926732] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11783.144434] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11783.155770] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11783.158648] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11788.257074] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11788.271032] Lustre: Skipped 2 previous similar messages [11789.235893] Lustre: server umount lustre-OST0001 complete [11793.395319] LustreError: 330864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11793.405720] LustreError: 330864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11797.486387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11797.491308] LustreError: Skipped 1 previous similar message [11797.496172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11797.513161] Lustre: Skipped 1 previous similar message [11797.528622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11798.506615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.618115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11803.632143] Lustre: Skipped 3 previous similar messages [11808.755976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11809.247593] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11809.389973] Lustre: server umount lustre-OST0000 complete [11813.856865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11813.866522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11813.881671] Lustre: Skipped 1 previous similar message [11818.977083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11818.987269] Lustre: Skipped 3 previous similar messages [11819.688201] Lustre: server umount lustre-MDT0000 complete [11824.556204] LustreError: 327889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789094346 with bad export cookie 10975796357175870748 [11824.581601] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11824.590145] LustreError: 327889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11825.178606] Lustre: server umount lustre-MDT0001 complete [11840.647291] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [11844.725753] Key type lgssc unregistered [11845.096925] LNet: 332855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11845.109550] LNetError: 332855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11845.144295] LNet: Removed LNI 192.168.202.152@tcp [11846.436746] Key type .llcrypt unregistered [11846.439357] Key type ._llcrypt unregistered [11898.555973] Key type ._llcrypt registered [11898.558811] Key type .llcrypt registered [11898.732702] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [11900.511411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11900.567607] alg: No test for adler32 (adler32-zlib) [11901.686552] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [11901.978349] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [11903.663973] Key type lgssc registered [11905.137538] Lustre: Echo OBD driver; http://www.lustre.org/ [11914.577477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11914.614061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11916.214162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11919.934874] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11926.502397] LustreError: 335149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11926.520305] LustreError: 335149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11930.086438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.408567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11934.995117] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11938.742720] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11946.714511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11947.340799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11955.492615] LustreError: 336637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11955.601945] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11960.810520] LustreError: 336639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11961.869060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11966.967592] LustreError: 336992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11966.996557] LustreError: 336992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11967.717291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11968.226223] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11973.645744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11978.543711] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [11989.967527] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11997.841302] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12012.516655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12012.522682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12012.530306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12014.561558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12014.571355] Lustre: Skipped 1 previous similar message [12014.572368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12014.588777] Lustre: Skipped 3 previous similar messages [12018.754085] Lustre: server umount lustre-MDT0000 complete [12019.695761] LustreError: 335150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12019.718087] LustreError: 335150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12024.028869] LustreError: 335130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789094546 with bad export cookie 1646772393533920896 [12024.030178] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12024.043984] LustreError: 335130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12024.862213] Lustre: server umount lustre-MDT0001 complete [12040.664706] Lustre: server umount lustre-OST0000 complete [12040.995775] Lustre: 334414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789094546/real 1789094546] req@ffff99c68e79f480 x1876001473762944/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789094562 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12041.030580] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12041.051533] Lustre: Skipped 1 previous similar message [12045.343171] Lustre: 334412:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789094551/real 1789094551] req@ffff99c68e79e300 x1876001473763200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789094567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12045.830790] Lustre: server umount lustre-OST0001 complete [12066.915225] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [12077.605086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12077.970240] LustreError: 340417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12078.042850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12082.865439] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12092.154240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12092.769488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12098.247843] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12101.989835] Lustre: 341558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12109.997685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12110.448362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12116.846933] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12117.619769] LustreError: 341911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12117.637745] LustreError: 341911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12124.159284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12126.900795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12132.377079] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12134.761823] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12143.165977] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12147.205786] Lustre: 343430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12152.801976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12152.816738] Lustre: Skipped 1 previous similar message [12152.817131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12152.825227] Lustre: Skipped 2 previous similar messages [12157.537452] Lustre: server umount lustre-MDT0000 complete [12157.922626] LustreError: 340417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.952203] LustreError: 340417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12162.408099] LustreError: 340398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789094684 with bad export cookie 1646772393533922604 [12162.408796] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12162.430374] LustreError: 340398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12163.103542] Lustre: server umount lustre-MDT0001 complete [12177.823595] Lustre: server umount lustre-OST0000 complete [12179.167204] Lustre: 334413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789094685/real 1789094685] req@ffff99c69d051880 x1876001473814528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789094701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12179.196041] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12179.211180] Lustre: Skipped 1 previous similar message [12181.791455] Lustre: server umount lustre-OST0001 complete [12186.131081] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12187.628292] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12189.519949] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 22:45:08 (1789094708) [12211.362254] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [12219.238245] loop0: detected capacity change from 0 to 1099511627776 [12219.821540] SGI XFS with ACLs, security attributes, quota, no debug enabled [12219.836935] *** VALIDATE xfs *** [12219.895658] XFS (loop0): Mounting V5 Filesystem [12219.934842] XFS (loop0): Ending clean mount [12221.698915] loop1: detected capacity change from 0 to 18691697672192 [12333.661485] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [12344.459661] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [12383.156907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12383.695929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12383.790154] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12384.001500] Lustre: lustre-MDT0000: new disk, initializing [12384.217942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12384.226013] Lustre: Skipped 1 previous similar message [12384.257302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12391.781414] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12404.439132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12404.657465] Lustre: 348565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12404.781700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12404.791721] Lustre: Skipped 1 previous similar message [12404.935932] Lustre: lustre-MDT0001: new disk, initializing [12405.124203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12405.141538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12411.219823] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12421.351564] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12428.597986] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12437.342726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12437.966582] Lustre: lustre-OST0000: new disk, initializing [12437.973250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12437.993645] Lustre: 350536:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12439.751808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12439.765058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12439.853598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12446.563520] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12457.328504] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12461.563450] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12461.745476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12467.015123] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12467.432315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12470.755623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12470.756704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12470.758530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12470.758537] Lustre: Skipped 1 previous similar message [12470.767083] LustreError: Skipped 1 previous similar message [12470.787505] Lustre: Skipped 1 previous similar message [12475.901941] Lustre: server umount lustre-OST0000 complete [12480.996306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12481.001157] Lustre: Skipped 2 previous similar messages [12490.212349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12490.231399] Lustre: Skipped 3 previous similar messages [12494.303446] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12494.749365] Lustre: server umount lustre-MDT0000 complete [12495.329445] LustreError: 348577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12495.367933] LustreError: 348577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12499.451871] LustreError: 348559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789095021 with bad export cookie 1646772393533924158 [12499.459956] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12499.462448] LustreError: 348559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12499.997722] Lustre: server umount lustre-MDT0001 complete [12513.216750] XFS (loop0): Unmounting Filesystem [12519.043038] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 22:50:37 (1789095037) [12526.552494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12527.038886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12527.043722] Lustre: Skipped 2 previous similar messages [12531.567668] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12539.564574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12544.522819] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12551.794336] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12557.448669] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12564.346519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12571.053343] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12579.078874] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12587.500633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12587.510751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12587.518824] Lustre: Skipped 2 previous similar messages [12587.519750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12593.262372] Lustre: server umount lustre-OST0000 complete [12603.782546] Lustre: server umount lustre-MDT0000 complete [12608.716691] LustreError: 355776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789095130 with bad export cookie 1646772393533925040 [12608.717396] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12608.736083] LustreError: 355776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12609.288953] Lustre: server umount lustre-MDT0001 complete [12623.405765] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [12627.099740] Key type lgssc unregistered [12627.439577] LNet: 356730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12627.467262] LNetError: 356730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12627.526310] LNet: Removed LNI 192.168.202.152@tcp [12628.649159] Key type .llcrypt unregistered [12628.651098] Key type ._llcrypt unregistered [12665.637381] Key type ._llcrypt registered [12665.639887] Key type .llcrypt registered [12665.706742] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [12667.212206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12667.266417] alg: No test for adler32 (adler32-zlib) [12668.351280] Lustre: Lustre: Build Version: 2.17.55_25_gfa4fa99 [12668.569034] LNet: Added LNI 192.168.202.152@tcp [8/256/0/180] [12670.257426] Key type lgssc registered [12671.366109] Lustre: Echo OBD driver; http://www.lustre.org/ [12679.171621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12679.187646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12680.431329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12684.293918] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12690.912864] LustreError: 358297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12690.937177] LustreError: 358297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12694.097841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12694.418927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12698.757249] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12706.525782] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12712.979834] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12719.918835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12720.210781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12728.117550] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12738.076287] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12750.307856] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12750.324807] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12750.330277] Lustre: Skipped 1 previous similar message [12751.612640] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [12753.700345] Lustre: server umount lustre-MDT0001 complete [12755.426752] LustreError: 358297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12755.446547] LustreError: 358297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12756.690514] LustreError: 358294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12760.548513] LustreError: 359301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12760.569049] LustreError: 359301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12763.760979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12763.836846] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12763.838947] Lustre: Found index 1 for lustre-MDT0001, updating log [12763.881065] Lustre: 358286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12764.292664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12768.568156] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12769.783561] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12769.839795] LustreError: 361800:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12769.850996] LustreError: 361800:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12769.875492] LustreError: 358289:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12769.939519] LustreError: 361808:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.152@tcp: cfg command failed: rc = -17 [12769.945576] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12769.945576] [12780.019652] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12780.022345] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12780.073784] Lustre: Skipped 1 previous similar message [12780.084873] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12782.299690] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [12782.311438] Lustre: Skipped 1 previous similar message [12784.926815] Lustre: server umount lustre-MDT0001 complete [12785.123922] LustreError: 359025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12785.135251] LustreError: 359025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12795.157106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12795.246179] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12795.251648] Lustre: Found index 1 for lustre-MDT0001, updating log [12795.291423] Lustre: 358286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12795.361334] LustreError: 359301:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12795.384423] LustreError: 359301:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12795.581887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12799.399493] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [12801.012897] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12805.152705] LustreError: 362970:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12805.166602] LustreError: 362970:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12805.194600] LustreError: 358289:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12805.217707] LustreError: 358289:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12805.388476] LustreError: 362977:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12805.399965] LustreError: 362977:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12805.415197] LustreError: 362977:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.152@tcp: cfg command failed: rc = -17 [12805.436892] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.152@tcp [12805.436892] [13104.860872] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.52@tcp (stopping) [13108.193193] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13108.194405] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13108.209123] Lustre: Skipped 1 previous similar message [13110.700335] Lustre: server umount lustre-MDT0001 complete [13113.327143] LustreError: 358294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13113.349977] LustreError: 358294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13121.403373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13121.596963] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13121.609162] Lustre: Found index 1 for lustre-MDT0001, updating log [13121.669121] Lustre: 358287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13121.982120] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13126.637368] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13127.162302] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13131.257465] LustreError: 371295:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13131.267300] LustreError: 371295:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13131.286100] LustreError: 358289:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13131.292749] LustreError: 358289:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13131.390340] LustreError: 371298:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13131.414658] LustreError: 371298:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.152@tcp: cfg command failed: rc = -114 [13131.425773] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13131.425773] [13449.700949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13449.704051] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13449.707963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13449.707978] Lustre: Skipped 3 previous similar messages [13449.753196] Lustre: Skipped 3 previous similar messages [13453.969800] Lustre: server umount lustre-MDT0000 complete [13454.819428] LustreError: 358293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13454.843632] LustreError: 358293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [13457.858598] LustreError: 358279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789095979 with bad export cookie 3907819762129525621 [13457.861471] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13457.874622] LustreError: 358279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13459.936962] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13467.050714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13467.062402] Lustre: Skipped 4 previous similar messages [13472.223346] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13472.405208] Lustre: server umount lustre-MDT0001 complete [13476.605783] Lustre: server umount lustre-OST0000 complete [13486.620384] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [13500.280146] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [13537.971493] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 23:07:37 (1789096057) [13543.597030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13543.894472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13543.925838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13544.003688] Lustre: lustre-MDT0000: new disk, initializing [13544.090452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13544.103053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13548.730924] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13559.192405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13559.364767] Lustre: 381843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13559.433607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13559.436402] Lustre: Skipped 1 previous similar message [13559.511687] Lustre: lustre-MDT0001: new disk, initializing [13559.588422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13559.617164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13559.633431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13564.264776] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13575.450307] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13582.505863] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13591.092351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13591.486652] Lustre: lustre-OST0000: new disk, initializing [13591.495547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13591.512536] Lustre: 383808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13591.645433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13597.340588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13597.355186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13597.666274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13600.110630] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_default_debug -1 all [13613.490617] Lustre: DEBUG MARKER: oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13623.507052] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13623.532245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13623.561845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13623.574275] Lustre: Skipped 1 previous similar message [13623.962784] Lustre: server umount lustre-MDT0000 complete [13628.387944] LustreError: 381853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13628.421345] LustreError: 381853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13628.991764] LustreError: 383807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789096151 with bad export cookie 3907819762129527574 [13628.998739] LustreError: 382647:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13629.007788] LustreError: MGC192.168.202.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13629.388876] Lustre: server umount lustre-MDT0001 complete [13645.306913] Lustre: server umount lustre-OST0000 complete [13681.170219] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing set_hostid [13694.405654] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing load_modules_local [13730.213796] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13386 sec ========= 23:10:48 (1789096248) [13732.941244] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:10:51 (1789096251) === [13737.456107] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:10:56 (1789096256) === [13782.348650] Lustre: DEBUG MARKER: oleg252-server.virtnet: executing unload_modules_local [13785.811629] Key type lgssc unregistered [13786.191964] LNet: 390176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13786.199864] LNetError: 390176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13786.229397] LNet: Removed LNI 192.168.202.152@tcp [13787.231937] Key type .llcrypt unregistered [13787.235755] Key type ._llcrypt unregistered