[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518307744 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002404] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007528] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010160] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.013014] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.022470] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027035] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] *** VALIDATE tmpfs *** [ 0.032204] *** VALIDATE proc *** [ 0.033289] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039039] Spectre V2 : User space: Vulnerable [ 0.040014] Speculative Store Bypass: Vulnerable [ 0.042793] debug: unmapping init [mem 0xffffffff92a59000-0xffffffff92a60fff] [ 0.044183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045656] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053323] rcu: Hierarchical SRCU implementation. [ 0.055500] smp: Bringing up secondary CPUs ... [ 0.056486] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060522] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.266274] node 0 deferred pages initialised in 199ms [ 0.270255] devtmpfs: initialized [ 0.271172] x86/mm: Memory block size: 128MB [ 0.273691] gcov: version magic: 0x41383552 [ 0.275172] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.276092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.277287] pinctrl core: initialized pinctrl subsystem [ 0.278201] [ 0.278728] ************************************************************* [ 0.279015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280021] ** ** [ 0.281014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.282017] ** ** [ 0.283016] ** This means that this kernel is built to expose internal ** [ 0.284019] ** IOMMU data structures, which may compromise security on ** [ 0.285018] ** your system. ** [ 0.286016] ** ** [ 0.287014] ** If you see this message and you are not debugging the ** [ 0.288016] ** kernel, report this immediately to your vendor! ** [ 0.289016] ** ** [ 0.290017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.291015] ************************************************************* [ 0.292586] NET: Registered protocol family 16 [ 0.293472] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.294068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.295070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297051] cpuidle: using governor menu [ 0.299909] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.303799] PCI: Using configuration type 1 for base access [ 0.306165] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.317133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.318028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.320090] cryptd: max_cpu_qlen set to 1000 [ 0.322265] ACPI: Added _OSI(Module Device) [ 0.323013] ACPI: Added _OSI(Processor Device) [ 0.324024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.325014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.328990] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.332560] ACPI: Interpreter enabled [ 0.333072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.334014] ACPI: Using IOAPIC for interrupt routing [ 0.335128] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.336395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.347741] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.348049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.349022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.350091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.352398] acpiphp: Slot [2] registered [ 0.353167] acpiphp: Slot [5] registered [ 0.354160] acpiphp: Slot [6] registered [ 0.355133] acpiphp: Slot [7] registered [ 0.356136] acpiphp: Slot [8] registered [ 0.357140] acpiphp: Slot [9] registered [ 0.358168] acpiphp: Slot [10] registered [ 0.359184] acpiphp: Slot [3] registered [ 0.360101] acpiphp: Slot [4] registered [ 0.361109] acpiphp: Slot [11] registered [ 0.362116] acpiphp: Slot [12] registered [ 0.363108] acpiphp: Slot [13] registered [ 0.364104] acpiphp: Slot [14] registered [ 0.365099] acpiphp: Slot [15] registered [ 0.366128] acpiphp: Slot [16] registered [ 0.367081] acpiphp: Slot [17] registered [ 0.368097] acpiphp: Slot [18] registered [ 0.369133] acpiphp: Slot [19] registered [ 0.370127] acpiphp: Slot [20] registered [ 0.371103] acpiphp: Slot [21] registered [ 0.372152] acpiphp: Slot [22] registered [ 0.373107] acpiphp: Slot [23] registered [ 0.374113] acpiphp: Slot [24] registered [ 0.375120] acpiphp: Slot [25] registered [ 0.376167] acpiphp: Slot [26] registered [ 0.377110] acpiphp: Slot [27] registered [ 0.378106] acpiphp: Slot [28] registered [ 0.379130] acpiphp: Slot [29] registered [ 0.380104] acpiphp: Slot [30] registered [ 0.381094] acpiphp: Slot [31] registered [ 0.382069] PCI host bridge to bus 0000:00 [ 0.383043] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.385034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.387043] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.390036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.393050] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.396042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.399273] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.403276] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.407743] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.419711] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.424115] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.427032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.430030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.433031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.436755] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.439887] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.444060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.448426] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.454030] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.467028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.473021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.483293] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.494025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.501023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.523035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.542047] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.553027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.568027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.601028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.612000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.621027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.629029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.652077] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.664036] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.673028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.682031] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.702025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.716000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.728022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.739024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.763029] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.774130] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.783030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.791039] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.815026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.829578] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.834533] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.837520] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.840581] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.843413] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.851152] iommu: Default domain type: Passthrough [ 0.853650] SCSI subsystem initialized [ 0.855188] ACPI: bus type USB registered [ 0.857227] usbcore: registered new interface driver usbfs [ 0.859127] usbcore: registered new interface driver hub [ 0.861162] usbcore: registered new device driver usb [ 0.863311] pps_core: LinuxPPS API ver. 1 registered [ 0.866020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.869101] PTP clock support registered [ 0.871158] EDAC MC: Ver: 3.0.0 [ 0.873185] PCI: Using ACPI for IRQ routing [ 0.876032] NetLabel: Initializing [ 0.878017] NetLabel: domain hash size = 128 [ 0.879017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.882134] NetLabel: unlabeled traffic allowed by default [ 0.885159] vgaarb: loaded [ 0.887219] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.888013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.893210] clocksource: Switched to clocksource kvm-clock [ 1.008605] VFS: Disk quotas dquot_6.6.0 [ 1.010815] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.014297] *** VALIDATE ramfs *** [ 1.016171] *** VALIDATE hugetlbfs *** [ 1.018678] pnp: PnP ACPI init [ 1.022850] pnp: PnP ACPI: found 6 devices [ 1.044172] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.048514] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.051346] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.053439] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.056394] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.059515] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.063325] NET: Registered protocol family 2 [ 1.066663] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.073433] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.078817] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.085341] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.089797] TCP: Hash tables configured (established 65536 bind 65536) [ 1.093900] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.097296] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.100560] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.103760] NET: Registered protocol family 1 [ 1.107712] RPC: Registered named UNIX socket transport module. [ 1.110166] RPC: Registered udp transport module. [ 1.111944] RPC: Registered tcp transport module. [ 1.113939] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.117132] NET: Registered protocol family 44 [ 1.119145] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.121821] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.124360] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.127157] PCI: CLS 0 bytes, default 64 [ 1.129212] Unpacking initramfs... [ 2.614709] debug: unmapping init [mem 0xffff944dbcc54000-0xffff944dbffbffff] [ 2.621695] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.624014] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.627701] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.093295] Initialise system trusted keyrings [ 3.095370] Key type blacklist registered [ 3.097884] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.106687] zbud: loaded [ 3.124089] *** VALIDATE nfs *** [ 3.125825] *** VALIDATE nfs4 *** [ 3.127504] pstore: using deflate compression [ 3.132186] Platform Keyring initialized [ 3.238034] NET: Registered protocol family 38 [ 3.239768] Key type asymmetric registered [ 3.240882] Asymmetric key parser 'x509' registered [ 3.242409] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.244977] io scheduler mq-deadline registered [ 3.246290] io scheduler kyber registered [ 3.247749] io scheduler bfq registered [ 3.249434] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.252795] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.255290] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.257309] ACPI: Power Button [PWRF] [ 3.261729] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.266906] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.280796] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.287143] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.301282] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.330889] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.358253] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.362711] Non-volatile memory driver v1.3 [ 3.364632] Linux agpgart interface v0.103 [ 3.392406] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.394618] vda: detected capacity change from 0 to 76677120 [ 3.406841] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.409154] vdb: detected capacity change from 0 to 1073741824 [ 3.420467] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.423443] vdc: detected capacity change from 0 to 2621440000 [ 3.437170] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.439299] vdd: detected capacity change from 0 to 2621440000 [ 3.452203] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.455431] vde: detected capacity change from 0 to 4294967296 [ 3.473177] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.475772] vdf: detected capacity change from 0 to 4294967296 [ 3.481696] libphy: Fixed MDIO Bus: probed [ 3.486883] usbcore: registered new interface driver usbserial_generic [ 3.489536] usbserial: USB Serial support registered for generic [ 3.491844] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.496735] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.498401] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.500737] mousedev: PS/2 mouse device common for all mice [ 3.504538] rtc_cmos 00:05: RTC can wake from S4 [ 3.508162] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.509700] rtc_cmos 00:05: registered as rtc0 [ 3.517210] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.520638] intel_pstate: CPU model not supported [ 3.524842] hid: raw HID events driver (C) Jiri Kosina [ 3.527200] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.530275] usbcore: registered new interface driver usbhid [ 3.530281] usbhid: USB HID core driver [ 3.530421] drop_monitor: Initializing network drop monitor service [ 3.531116] Initializing XFRM netlink socket [ 3.545054] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.545333] NET: Registered protocol family 10 [ 3.551572] Segment Routing with IPv6 [ 3.553321] NET: Registered protocol family 17 [ 3.555742] mpls_gso: MPLS GSO support [ 3.561733] RAS: Correctable Errors collector initialized. [ 3.564330] AVX version of gcm_enc/dec engaged. [ 3.566310] AES CTR mode by8 optimization enabled [ 3.640049] sched_clock: Marking stable (3640020882, 0)->(4601770816, -961749934) [ 3.642632] registered taskstats version 1 [ 3.643902] Loading compiled-in X.509 certificates [ 3.645680] zswap: loaded using pool lzo/zbud [ 3.666795] Key type big_key registered [ 3.678672] Key type encrypted registered [ 3.679981] ima: No TPM chip found, activating TPM-bypass! [ 3.681468] ima: Allocated hash algorithm: sha1 [ 3.682612] ima: No architecture policies found [ 3.683857] evm: Initialising EVM extended attributes: [ 3.685101] evm: security.selinux [ 3.685857] evm: security.ima [ 3.686551] evm: security.capability [ 3.687355] evm: HMAC attrs: 0x1 [ 3.689244] rtc_cmos 00:05: setting system clock to 2026-09-03 12:07:54 UTC (1788437274) [ 3.693388] debug: unmapping init [mem 0xffffffff93a03000-0xffffffff93bfffff] [ 3.695779] debug: unmapping init [mem 0xffffffff92782000-0xffffffff92a58fff] [ 3.704068] Write protecting the kernel read-only data: 28672k [ 3.708981] debug: unmapping init [mem 0xffffffff90e03000-0xffffffff90ffffff] [ 3.712019] debug: unmapping init [mem 0xffffffff91714000-0xffffffff917fffff] [ 3.745493] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.751718] systemd[1]: Detected virtualization kvm. [ 3.753425] systemd[1]: Detected architecture x86-64. [ 3.754867] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.780292] systemd[1]: No hostname configured. [ 3.782095] systemd[1]: Set hostname to . [ 3.783720] random: systemd: uninitialized urandom read (16 bytes read) [ 3.786343] systemd[1]: Initializing machine ID from random generator. [ 3.926087] random: systemd: uninitialized urandom read (16 bytes read) [ 3.929167] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.933526] random: systemd: uninitialized urandom read (16 bytes read) [ 3.935661] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.938705] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.565372] device-mapper: uevent: version 1.0.3 [ 4.568071] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.224439] virtio_net virtio0 ens2: renamed from eth0 [ 5.270138] random: fast init done [ 5.379157] scsi host0: ata_piix [ 5.394527] scsi host1: ata_piix [ 5.404186] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.406898] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.918366] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.198410] random: crng init done [ 10.199968] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.514878] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.557278] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.823648] SELinux: Disabled at runtime. [ 11.881539] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.890690] systemd[1]: Detected virtualization kvm. [ 11.892153] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.368156] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.371155] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.378959] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.382120] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.384967] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.392131] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.399097] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ 12.417700] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.825519] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.117109] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.132682] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.208594] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.225108] EDAC sbridge: Ver: 1.1.2 [ 14.816302] Key type dns_resolver registered [ 15.088469] NFS: Registering the id_resolver key type [ 15.090554] Key type id_resolver registered [ 15.091852] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg201-server login: [ 57.349968] libcfs: loading out-of-tree module taints kernel. [ 57.387461] Key type ._llcrypt registered [ 57.389729] Key type .llcrypt registered [ 57.505379] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 68.249240] hrtimer: interrupt took 14788114 ns [ 80.619225] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 82.727585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.740168] alg: No test for adler32 (adler32-zlib) [ 84.228900] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 85.343880] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 87.231611] Key type lgssc registered [ 89.105398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.933246] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.065089] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 170.572487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 170.637936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.183647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.216690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.302946] Lustre: lustre-MDT0000: new disk, initializing [ 172.443458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.462726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.839962] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.439471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.525037] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.583993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 194.586958] Lustre: Skipped 1 previous similar message [ 194.646991] Lustre: lustre-MDT0001: new disk, initializing [ 194.691094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.722371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 194.749640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.694960] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.759959] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 214.314878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.588607] Lustre: lustre-OST0000: new disk, initializing [ 214.600931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 214.615136] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.681465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.490387] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.312744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.348270] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.390088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.303157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.478711] Lustre: lustre-OST0001: new disk, initializing [ 237.488966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.495810] Lustre: 9516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.571198] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.334039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.372234] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.490218] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 246.273903] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 259.574839] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 270.331672] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.559402] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.974724] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing yml_node [ 289.326067] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 291.993613] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 294.342439] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 296.119690] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:12:44 EDT 2026 [ 314.693923] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 316.258316] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.771220] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.773594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.784268] Lustre: Skipped 2 previous similar messages [ 332.796504] Lustre: Skipped 1 previous similar message [ 337.891708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 337.909321] Lustre: Skipped 5 previous similar messages [ 338.927685] Lustre: server umount lustre-MDT0000 complete [ 342.913930] LustreError: 7454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437613 with bad export cookie 4795344020030046549 [ 342.914785] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 342.922257] LustreError: 7454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.008369] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.009474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.014965] Lustre: Skipped 2 previous similar messages [ 343.040269] Lustre: Skipped 1 previous similar message [ 347.039968] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.043895] Lustre: Skipped 1 previous similar message [ 352.224112] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.224914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.254836] LustreError: 6513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 352.271035] Lustre: Skipped 1 previous similar message [ 353.249915] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.276635] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 357.343142] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 357.568322] Lustre: server umount lustre-MDT0001 complete [ 361.916946] Lustre: server umount lustre-OST0000 complete [ 366.336221] Lustre: server umount lustre-OST0001 complete [ 374.190258] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 385.208277] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 428.668629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 429.177181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 429.216907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 429.310395] Lustre: lustre-MDT0000: new disk, initializing [ 429.450087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.474348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 435.238096] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 447.971307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.093605] Lustre: 15073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 448.148838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 448.154654] Lustre: Skipped 1 previous similar message [ 448.232354] Lustre: lustre-MDT0001: new disk, initializing [ 448.396706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 448.455687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 448.472237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 453.477686] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 463.722290] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 469.235734] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 475.752550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 476.057539] Lustre: lustre-OST0000: new disk, initializing [ 476.064285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 476.070773] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 476.171234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.451265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 477.460206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 477.579634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 482.061741] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 492.440550] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 497.725446] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 498.091787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 503.217862] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 503.551751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 508.396604] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 508.413138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 508.418179] Lustre: Skipped 1 previous similar message [ 508.441438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 508.441448] Lustre: Skipped 2 previous similar messages [ 512.350810] Lustre: server umount lustre-OST0000 complete [ 518.628206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.645319] Lustre: Skipped 1 previous similar message [ 523.934894] Lustre: server umount lustre-MDT0000 complete [ 527.575736] LustreError: 15869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437798 with bad export cookie 4795344020030048082 [ 527.580798] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.584804] LustreError: 15869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 528.015758] Lustre: server umount lustre-MDT0001 complete [ 532.916571] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 08:16:41 (1788437801) [ 540.879411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.348882] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.492961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 545.658332] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 546.784552] LustreError: 19307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.914277] LustreError: 19306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.519238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.302809] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 570.108124] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 577.125504] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.914888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.402662] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 602.060596] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 606.137973] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 615.401979] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.413288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 615.419733] Lustre: Skipped 7 previous similar messages [ 620.412736] Lustre: server umount lustre-MDT0000 complete [ 620.528963] LustreError: 20033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.554631] LustreError: 20033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 624.507198] LustreError: 21835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437895 with bad export cookie 4795344020030048957 [ 624.510472] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.512378] LustreError: 21835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 624.775528] Lustre: server umount lustre-MDT0001 complete [ 637.455984] Lustre: server umount lustre-OST0000 complete [ 665.460395] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 677.828753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 678.368258] LustreError: 23701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.536311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 678.555128] Lustre: Skipped 2 previous similar messages [ 684.342969] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 693.977439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.674923] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 703.758972] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 711.706461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.168309] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 729.091586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 729.435618] Lustre: lustre-OST0001: new disk, initializing [ 729.445300] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 729.459972] Lustre: 26017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 730.851543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 730.885155] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 731.044516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 736.616872] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 749.088120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 753.523823] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 766.439228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 766.451305] LustreError: Skipped 1 previous similar message [ 766.458619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.485301] Lustre: Skipped 2 previous similar messages [ 766.499895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 772.273764] Lustre: server umount lustre-MDT0000 complete [ 776.656738] LustreError: 26016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438047 with bad export cookie 4795344020030049811 [ 776.671071] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 776.672579] LustreError: 26016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 777.260609] Lustre: server umount lustre-MDT0001 complete [ 791.654553] Lustre: server umount lustre-OST0000 complete [ 792.484466] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438047/real 1788438047] req@ffff944d04070000 x1875312489255424/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788438063 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 792.534588] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.558454] Lustre: Skipped 3 previous similar messages [ 797.497802] Lustre: server umount lustre-OST0001 complete [ 822.983510] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 834.911045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.225943] LustreError: 29036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.250859] LustreError: 29036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 835.327712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 835.330832] Lustre: Skipped 3 previous similar messages [ 840.475870] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 850.612921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 850.914750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 850.917917] Lustre: Skipped 7 previous similar messages [ 856.469183] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 859.826662] Lustre: 30182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 867.389805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 875.272760] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 884.452546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 891.523616] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 900.357800] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 904.891740] Lustre: 32061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 912.353627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 912.364906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 916.946455] Lustre: server umount lustre-MDT0000 complete [ 920.385883] LustreError: 30563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438191 with bad export cookie 4795344020030051106 [ 920.393585] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 920.403734] LustreError: 30563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 920.546617] LustreError: 30034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.586228] LustreError: 30034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 920.851638] Lustre: server umount lustre-MDT0001 complete [ 926.807987] Lustre: server umount lustre-OST0000 complete [ 930.701168] Lustre: server umount lustre-OST0001 complete [ 939.380574] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 08:23:27 (1788438207) [ 972.546716] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 984.537261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.467230] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1000.834499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.421689] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1009.882961] Lustre: 36014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1017.625459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1026.362453] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1035.204532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1041.713114] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1051.379926] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1055.755686] Lustre: 37896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1061.356806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.368223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1061.372774] Lustre: Skipped 7 previous similar messages [ 1061.387848] Lustre: Skipped 10 previous similar messages [ 1067.701854] Lustre: server umount lustre-MDT0000 complete [ 1071.616522] LustreError: 35875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1071.662553] LustreError: 35875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 1072.071609] LustreError: 37898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438342 with bad export cookie 4795344020030052212 [ 1072.079187] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1072.080978] LustreError: 37898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1072.474850] Lustre: server umount lustre-MDT0001 complete [ 1087.948133] Lustre: server umount lustre-OST0000 complete [ 1102.167766] Lustre: server umount lustre-OST0001 complete [ 1112.442859] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 08:26:20 (1788438380) [ 1135.887697] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 1147.634126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.304340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.309994] Lustre: Skipped 7 previous similar messages [ 1154.549057] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1165.914158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.302642] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1173.149732] Lustre: 41211:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1179.765117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1186.774994] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1195.936626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1202.890118] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1211.558455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1216.378117] Lustre: 43088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1217.496864] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1217.512934] Lustre: Skipped 1 previous similar message [ 1222.123186] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.145698] Lustre: Skipped 2 previous similar messages [ 1226.868290] Lustre: server umount lustre-MDT0000 complete [ 1231.787667] LustreError: 41591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438502 with bad export cookie 4795344020030053318 [ 1231.794508] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1231.798447] LustreError: 41591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1246.176045] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1246.523820] Lustre: server umount lustre-MDT0001 complete [ 1256.667384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.929043] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff944e3d53ce00 x1875312489416320/t0(0) o250->MGC192.168.202.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1262.304280] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1271.805893] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1272.865296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.454415] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1278.464602] Lustre: Skipped 1 previous similar message [ 1279.443945] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1289.103288] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1291.863110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1300.610322] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1302.522755] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1309.155441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1314.698388] Lustre: server umount lustre-MDT0000 complete [ 1318.821398] LustreError: 40042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438589 with bad export cookie 4795344020030054466 [ 1318.826526] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1318.836987] LustreError: 40042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1319.226742] Lustre: server umount lustre-MDT0001 complete [ 1334.223882] Lustre: server umount lustre-OST0000 complete [ 1336.039400] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438590/real 1788438590] req@ffff944d04122300 x1875312489452672/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788438606 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1339.008968] Lustre: server umount lustre-OST0001 complete [ 1348.482159] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 08:30:17 (1788438617) [ 1368.360223] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 1380.139156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.452973] LustreError: 47703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.474880] LustreError: 47703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [ 1385.908203] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1394.920712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1400.971625] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1404.400776] Lustre: 48854:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1412.434682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1420.050401] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1428.924973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1436.238437] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1444.894775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1450.276814] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1461.729100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1461.739520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.744669] Lustre: Skipped 25 previous similar messages [ 1463.228254] Lustre: server umount lustre-MDT0000 complete [ 1466.970028] LustreError: 47684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438737 with bad export cookie 4795344020030055285 [ 1466.977240] LustreError: 47684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1466.977987] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.265476] Lustre: server umount lustre-MDT0001 complete [ 1482.087558] Lustre: server umount lustre-OST0000 complete [ 1496.366407] Lustre: server umount lustre-OST0001 complete [ 1507.228386] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 08:32:55 (1788438775) [ 1511.015880] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1513.772736] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 08:33:01 (1788438781) [ 1527.354941] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 1539.477206] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 1565.026055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.355949] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1565.485498] Lustre: lustre-MDT0000: new disk, initializing [ 1565.800537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1571.325955] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1584.518324] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1594.584663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1594.909829] Lustre: lustre-OST0000: new disk, initializing [ 1594.913914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1594.917463] Lustre: Skipped 1 previous similar message [ 1594.923816] Lustre: 54930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1596.327109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1596.349642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1596.423038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1604.001591] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1614.327256] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1617.471916] Lustre: DEBUG MARKER: create test files [ 1657.827954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.849238] Lustre: Skipped 13 previous similar messages [ 1663.041847] Lustre: server umount lustre-OST0000 complete [ 1667.369361] Lustre: server umount lustre-MDT0000 complete [ 1681.839482] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 1685.030517] Key type lgssc unregistered [ 1685.375803] LNet: 56564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.382579] LNetError: 56564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.394625] LNet: Removed LNI 192.168.202.101@tcp [ 1686.737162] Key type .llcrypt unregistered [ 1686.744692] Key type ._llcrypt unregistered [ 1723.108500] Key type ._llcrypt registered [ 1723.112195] Key type .llcrypt registered [ 1723.255677] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 1724.996860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1725.083764] alg: No test for adler32 (adler32-zlib) [ 1726.313756] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1726.632464] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 1728.345516] Key type lgssc registered [ 1729.969187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1739.956542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1740.009464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.751916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1747.073366] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1756.396277] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1764.718539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1765.068620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.180786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1767.278731] Lustre: 59298:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1767.284829] Lustre: 59298:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1767.295340] Lustre: 59298:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1767.300637] Lustre: 59298:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1767.307069] Lustre: 59298:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1767.317483] Lustre: 59298:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1772.003722] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1780.942444] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1787.457555] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1794.623071] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1803.235382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.269268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1808.353529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1808.915945] Lustre: server umount lustre-OST0000 complete [ 1813.123477] Lustre: server umount lustre-MDT0000 complete [ 1827.603294] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 1830.381986] Key type lgssc unregistered [ 1830.706838] LNet: 60908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.711524] LNetError: 60908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.731191] LNet: Removed LNI 192.168.202.101@tcp [ 1831.816528] Key type .llcrypt unregistered [ 1831.822256] Key type ._llcrypt unregistered [ 1869.543912] Key type ._llcrypt registered [ 1869.546346] Key type .llcrypt registered [ 1869.632597] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 1870.583793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1870.681708] alg: No test for adler32 (adler32-zlib) [ 1872.022657] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1872.438590] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 1874.207177] Key type lgssc registered [ 1875.507850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1883.577170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1883.604786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.247817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.494995] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1898.906828] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1907.471658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1908.010532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.095616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1911.156871] Lustre: 63753:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1911.164900] Lustre: 63753:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1911.169884] Lustre: 63753:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1911.184969] Lustre: 63753:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1911.203500] Lustre: 63753:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1911.212376] Lustre: 63753:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1914.709932] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 1923.137852] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1930.555584] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1972.707820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1972.726217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.760372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1975.766894] Lustre: server umount lustre-OST0000 complete [ 1980.217218] Lustre: server umount lustre-MDT0000 complete [ 1996.204397] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 1999.449903] Key type lgssc unregistered [ 1999.741053] LNet: 65338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.758456] LNetError: 65338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1999.793566] LNet: Removed LNI 192.168.202.101@tcp [ 2000.846952] Key type .llcrypt unregistered [ 2000.851475] Key type ._llcrypt unregistered [ 2022.087814] Key type ._llcrypt registered [ 2022.090227] Key type .llcrypt registered [ 2022.183709] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 2042.485689] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 2044.198245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2044.302748] alg: No test for adler32 (adler32-zlib) [ 2045.338355] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2045.628736] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 2047.423223] Key type lgssc registered [ 2048.959048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2094.507561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2094.544924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.909836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2095.964866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2096.119526] Lustre: lustre-MDT0000: new disk, initializing [ 2096.257943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.277434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2101.496565] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2113.189741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.391032] Lustre: 68489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2113.470789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2113.473693] Lustre: Skipped 1 previous similar message [ 2113.545867] Lustre: lustre-MDT0001: new disk, initializing [ 2113.649233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2113.717476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2113.728065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2118.592945] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2128.801189] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2136.291655] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2143.802154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.168330] Lustre: lustre-OST0000: new disk, initializing [ 2144.171921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2144.175947] Lustre: 70459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2144.266781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.067865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2152.086964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2152.256478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2152.522574] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2163.818992] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2170.915287] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2171.277994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2178.150923] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2178.443834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2183.136392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2183.145120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.161310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2183.171825] Lustre: Skipped 1 previous similar message [ 2183.191453] Lustre: Skipped 1 previous similar message [ 2186.921251] Lustre: server umount lustre-OST0000 complete [ 2193.378106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.403797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.411703] Lustre: Skipped 1 previous similar message [ 2198.502037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2198.508851] Lustre: Skipped 1 previous similar message [ 2203.616950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.629963] Lustre: Skipped 1 previous similar message [ 2205.151130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2205.439929] Lustre: server umount lustre-MDT0000 complete [ 2209.380175] LustreError: 68481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439480 with bad export cookie 2799035926911209927 [ 2209.403589] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.662493] Lustre: server umount lustre-MDT0001 complete [ 2218.716649] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 08:44:47 (1788439487) [ 2232.599475] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 2281.429994] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 2294.332125] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 2333.049097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2333.230519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2333.250113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2333.306942] Lustre: lustre-MDT0000: new disk, initializing [ 2333.407512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.424529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2338.433673] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2349.277307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.430352] Lustre: 76485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2349.483858] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2349.492338] Lustre: Skipped 1 previous similar message [ 2349.567865] Lustre: lustre-MDT0001: new disk, initializing [ 2349.697624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2349.735807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2349.759336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2354.472753] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2364.785241] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2372.059415] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2380.129926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2380.515558] Lustre: lustre-OST0000: new disk, initializing [ 2380.522546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2380.532510] Lustre: 78453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2380.651499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.041566] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2388.515191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2388.525697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2388.684461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2399.310836] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2405.532407] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2405.746710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2411.341576] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2411.632142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2414.565229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.582957] Lustre: Skipped 1 previous similar message [ 2414.594232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2414.595154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2414.629055] LustreError: Skipped 1 previous similar message [ 2419.169179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2419.180762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.805130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.823796] Lustre: Skipped 2 previous similar messages [ 2428.383172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2428.826049] Lustre: server umount lustre-OST0000 complete [ 2435.044705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.058549] Lustre: Skipped 1 previous similar message [ 2445.280779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.290340] Lustre: Skipped 6 previous similar messages [ 2447.327283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2447.530814] Lustre: server umount lustre-MDT0000 complete [ 2449.380891] LustreError: 79035:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.397834] LustreError: 79035:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2451.222815] LustreError: 76477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439722 with bad export cookie 2799035926911210809 [ 2451.226197] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2451.232657] LustreError: 76477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2451.485429] Lustre: server umount lustre-MDT0001 complete [ 2461.594509] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 08:48:49 (1788439729) [ 2469.299041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2470.073710] LustreError: 80870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.198876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.490450] LustreError: 80871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2475.628040] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2479.586477] LustreError: 80870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.705900] LustreError: 80871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.340778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.270847] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2499.862731] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2507.686235] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2515.342547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2515.820298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.823851] Lustre: Skipped 1 previous similar message [ 2522.885482] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2532.272837] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2536.938579] Lustre: 83543:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 30f07545-5448-4f8d-b382-b172858f28c6 at adminstrative request [ 2539.373362] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.399485] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2549.744254] Lustre: MGS: Client 30f07545-5448-4f8d-b382-b172858f28c6 (at 0@lo) reconnecting [ 2549.749258] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x26d82c58791958ab to 0x26d82c5879195b4b [ 2549.776287] Lustre: MGC192.168.202.101@tcp: Connection restored to 0@lo (at 0@lo) [ 2576.145759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2576.339751] Lustre: lustre-OST0001: new disk, initializing [ 2576.343692] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2576.348845] Lustre: 83935:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2584.442411] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2586.759678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2586.769708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2586.843957] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2595.031355] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2602.464711] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2602.469116] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.485892] LustreError: Skipped 1 previous similar message [ 2602.487692] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2602.523800] Lustre: Skipped 1 previous similar message [ 2604.933789] Lustre: server umount lustre-OST0001 complete [ 2607.586661] LustreError: 82691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.622989] LustreError: 82691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2611.168729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.492832] Lustre: server umount lustre-OST0000 complete [ 2622.944217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.976378] Lustre: Skipped 3 previous similar messages [ 2625.584964] Lustre: server umount lustre-MDT0000 complete [ 2628.065380] LustreError: 80865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.081655] LustreError: 80865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2629.976630] LustreError: 82714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439900 with bad export cookie 2799035926911212363 [ 2629.999644] LustreError: 82714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2630.003539] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2630.493267] Lustre: server umount lustre-MDT0001 complete [ 2641.240654] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 08:51:49 (1788439909) [ 2643.140750] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2645.470949] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 08:51:53 (1788439913) [ 2647.314995] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2649.399731] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 08:51:57 (1788439917) [ 2651.235311] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2653.129884] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 08:52:01 (1788439921) [ 2654.995543] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2657.204719] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 08:52:05 (1788439925) [ 2659.364179] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2666.265996] loop: module loaded [ 2666.298523] loop0: detected capacity change from 0 to 4294967296 [ 2666.391672] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2668.250695] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2672.203767] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2678.418295] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2708.923887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.969708] LustreError: 88063:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2708.975129] LustreError: 88063:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2708.980640] LustreError: 88063:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2708.986981] LustreError: 88063:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2731.082975] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 2746.005112] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 2786.527072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.749930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2786.771278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2786.837286] Lustre: lustre-MDT0000: new disk, initializing [ 2786.947526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.952673] Lustre: Skipped 1 previous similar message [ 2786.964096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2792.257266] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2806.416540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.647805] Lustre: 90945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2807.178904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2814.471715] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2825.852356] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2832.287689] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2843.194235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2843.955710] Lustre: 92913:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2852.971616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2852.985101] Lustre: Skipped 1 previous similar message [ 2852.987759] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2853.188373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2855.419223] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2870.483464] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2876.573657] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2876.823337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2881.801961] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2882.178820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2884.064136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2884.074944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.077222] LustreError: Skipped 1 previous similar message [ 2884.081099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2884.081109] Lustre: Skipped 6 previous similar messages [ 2884.120800] Lustre: Skipped 1 previous similar message [ 2898.399252] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2898.547787] Lustre: server umount lustre-OST0000 complete [ 2904.544353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2908.607460] Lustre: server umount lustre-MDT0000 complete [ 2909.664939] LustreError: 92026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2909.676811] LustreError: 92026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2912.352829] LustreError: 90936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440183 with bad export cookie 2799035926911212979 [ 2912.360335] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2912.364376] LustreError: 90936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2912.625369] Lustre: server umount lustre-MDT0001 complete [ 2923.137120] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 08:56:31 (1788440191) [ 2930.260061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.035466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2931.047183] Lustre: Skipped 2 previous similar messages [ 2936.769460] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2946.034340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2951.398363] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2960.789177] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2968.322294] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2977.044728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2986.980514] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 2996.664471] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3002.574608] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3002.853603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3008.055309] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3008.356185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3015.680637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3015.863886] Lustre: lustre-OST0001: new disk, initializing [ 3015.870740] Lustre: Skipped 2 previous similar messages [ 3015.888248] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3015.898611] Lustre: Skipped 3 previous similar messages [ 3015.906677] Lustre: 98703:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3021.457445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3021.468225] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3021.618883] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3023.957931] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3033.906691] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3038.720552] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3038.942420] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3044.094319] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3044.342736] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3064.786852] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3091.608968] Lustre: Failing over lustre-MDT0000 [ 3092.193396] Lustre: server umount lustre-MDT0000 complete [ 3092.844627] LustreError: 95325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3092.861723] LustreError: 95325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 3093.483334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.502091] Lustre: Skipped 4 previous similar messages [ 3109.856552] Lustre: 66713:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440364/real 1788440364] req@0000000010e6bfc8 x1875314544506240/t0(0) o400->MGC192.168.202.101@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788440380 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3109.879112] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3119.563831] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3119.566205] LDISKFS-fs (dm-0): recovery complete [ 3119.572670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.101797] LustreError: 66709:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000022e23e27 x1875314544514048/t0(0) o250->MGC192.168.202.101@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3120.568474] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3123.553810] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3123.555647] Lustre: 95326:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3125.084818] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3125.733346] Lustre: 95330:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3125.745712] Lustre: 95330:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 3125.764240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3125.860457] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3126.183175] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3126.528276] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3126.554801] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 3126.912442] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3126.927966] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 3126.953093] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3126.966840] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 3127.623918] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3127.647777] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 3127.975137] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3127.985265] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 3128.953595] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3128.960705] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 5 previous similar messages [ 3129.927023] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3129.930477] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 3130.263162] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3130.269327] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 3133.113678] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3133.126690] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 3134.080388] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3134.094787] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 3134.418953] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3134.427035] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 3141.175322] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3141.201580] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 24 previous similar messages [ 3142.188293] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3142.201207] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 3142.527098] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3142.533188] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 3157.475401] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3157.507378] Lustre: 101555:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 3158.464160] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3158.488206] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 3158.815589] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3158.821656] LustreError: 101555:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 3183.144229] Lustre: 101555:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3183.154280] Lustre: 101555:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3183.173305] Lustre: 101555:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 06efa31b-ed85-4db9-967b-70462ffe8548@192.168.202.1@tcp [ 3183.186133] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3183.208380] Lustre: 101555:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@000000005a93a52e x1875314522264320/t0(8589934775) o101->06efa31b-ed85-4db9-967b-70462ffe8548@192.168.202.1@tcp:239/0 lens 592/0 e 0 to 0 dl 1788440464 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3183.253758] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3183.260604] Lustre: lustre-MDT0000: Denying connection for new client 06efa31b-ed85-4db9-967b-70462ffe8548 (at 192.168.202.1@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3183.290264] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3183.349403] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3183.351224] Lustre: Skipped 2 previous similar messages [ 3183.357369] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3183.401802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:46 to 0x2c0000401:545) [ 3183.414518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 3199.457789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3199.466797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3199.471852] Lustre: Skipped 7 previous similar messages [ 3212.255245] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3212.546652] Lustre: server umount lustre-OST0000 complete [ 3223.021699] LustreError: 97153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.032765] LustreError: 97153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 3231.199130] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3231.423290] Lustre: server umount lustre-OST0001 complete [ 3238.368401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.383975] Lustre: Skipped 5 previous similar messages [ 3241.536960] Lustre: server umount lustre-MDT0000 complete [ 3245.156402] LustreError: 95310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440515 with bad export cookie 2799035926911264191 [ 3245.163591] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3245.170366] LustreError: 95310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3245.417799] Lustre: server umount lustre-MDT0001 complete [ 3255.786811] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:02:04 (1788440524) [ 3264.153893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.159816] Lustre: *** cfs_fail_loc=197, val=0*** [ 3279.649204] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3284.767950] LustreError: 103545:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.101@tcp: failed processing log, type 4: rc = -110 [ 3289.930700] Lustre: *** cfs_fail_loc=197, val=0*** [ 3289.947349] Lustre: Skipped 65 previous similar messages [ 3310.559905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.571951] Lustre: Skipped 4 previous similar messages [ 3317.442315] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3325.638769] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3328.206950] Lustre: server umount lustre-OST0000 complete [ 3339.168573] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 09:03:27 (1788440607) [ 3361.663656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.788928] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3377.820821] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3382.791641] LustreError: 105197:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3382.802068] LustreError: 105197:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3382.810478] LustreError: 105197:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 3383.602890] Lustre: server umount lustre-MDT0000 complete [ 3426.575983] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 3442.118428] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 3484.007724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.339948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3484.375313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3484.471794] Lustre: lustre-MDT0000: new disk, initializing [ 3484.587447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3490.538260] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3502.959278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.170859] Lustre: 109393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3503.594272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3508.919555] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3521.931312] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3531.178122] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3540.201586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3540.500272] Lustre: 111365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3547.296289] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3549.442978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3557.435795] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3562.566905] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3562.853916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3568.356983] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3568.789915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3575.270495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.271140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3575.291228] Lustre: Skipped 1 previous similar message [ 3575.295071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3575.295080] Lustre: Skipped 12 previous similar messages [ 3575.318280] LustreError: Skipped 1 previous similar message [ 3577.205408] Lustre: server umount lustre-OST0000 complete [ 3587.766785] Lustre: server umount lustre-MDT0000 complete [ 3590.632778] LustreError: 109401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.659538] LustreError: 109401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3592.265844] LustreError: 109386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440863 with bad export cookie 2799035926911274831 [ 3592.281521] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3592.287307] LustreError: 109386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3592.699443] Lustre: server umount lustre-MDT0001 complete [ 3603.316567] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:07:51 (1788440871) [ 3613.784303] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 3617.844822] Key type lgssc unregistered [ 3618.219761] LNet: 113851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3618.225142] LNetError: 113851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3618.235875] LNet: Removed LNI 192.168.202.101@tcp [ 3619.409775] Key type .llcrypt unregistered [ 3619.415202] Key type ._llcrypt unregistered [ 3647.248537] Key type ._llcrypt registered [ 3647.252332] Key type .llcrypt registered [ 3647.481049] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 3664.897897] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 3665.960430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3666.116207] alg: No test for adler32 (adler32-zlib) [ 3667.141744] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3667.364469] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 3669.031250] Key type lgssc registered [ 3670.198472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3715.565681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3715.597905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.904333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3716.924405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3716.995985] Lustre: lustre-MDT0000: new disk, initializing [ 3717.111783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.135347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3721.812902] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3732.940337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.158405] Lustre: lustre-OST0000: new disk, initializing [ 3733.161581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3733.164551] Lustre: Skipped 1 previous similar message [ 3733.172477] Lustre: 118030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3733.227426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.335711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3740.348038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3740.400087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3740.564205] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3752.213976] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3752.482682] Lustre: lustre-OST0001: new disk, initializing [ 3752.500227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3752.512594] Lustre: 118895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3752.671893] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3760.722313] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3761.851392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3761.858625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3761.886722] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3772.398223] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3772.408615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.413276] Lustre: Skipped 1 previous similar message [ 3772.437306] Lustre: Skipped 1 previous similar message [ 3776.545486] Lustre: server umount lustre-MDT0000 complete [ 3784.764671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3790.652288] LustreError: 117208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441061 with bad export cookie 2210287416384551608 [ 3790.663286] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3790.667734] LustreError: 117208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3790.876674] Lustre: server umount lustre-OST0000 complete [ 3795.388933] Lustre: server umount lustre-OST0001 complete [ 3805.138411] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 09:11:13 (1788441073) [ 3831.420491] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 09:11:39 (1788441099) [ 3843.983096] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 3876.200494] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:12:24 (1788441144) [ 3920.709871] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 3931.907987] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 3969.768252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.069644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3970.105304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3970.176948] Lustre: lustre-MDT0000: new disk, initializing [ 3970.256746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.297265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3974.815968] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 3986.042629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3986.234928] Lustre: 126655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3986.324479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3986.332023] Lustre: Skipped 1 previous similar message [ 3986.397605] Lustre: lustre-MDT0001: new disk, initializing [ 3986.476679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3986.518894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3986.526740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3991.478919] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4001.557669] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4008.412462] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4016.007251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4016.388611] Lustre: lustre-OST0000: new disk, initializing [ 4016.392889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4016.412795] Lustre: 128622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4016.525380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4022.352942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4022.359281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4022.505258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4024.397451] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4035.269157] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4041.391549] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4041.697853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4047.763798] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4048.076238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4053.494340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4053.495753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.502026] LustreError: Skipped 1 previous similar message [ 4053.514118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4053.523027] Lustre: Skipped 1 previous similar message [ 4053.532549] Lustre: Skipped 1 previous similar message [ 4056.657488] Lustre: server umount lustre-OST0000 complete [ 4063.207568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4063.219691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.237873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.276447] Lustre: server umount lustre-MDT0000 complete [ 4071.784607] LustreError: 126646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441342 with bad export cookie 2210287416384552826 [ 4071.793042] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4071.794692] LustreError: 126646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4072.393531] Lustre: server umount lustre-MDT0001 complete [ 4083.343399] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:15:50 (1788441350) [ 4091.214541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.114347] LustreError: 131037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.145591] LustreError: 131037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4092.271114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4097.506711] LustreError: 131038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.041373] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4101.602596] LustreError: 131037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.731473] LustreError: 131038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4108.331071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4114.738694] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4123.827638] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4130.515733] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4138.219854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4138.553166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.560865] Lustre: Skipped 1 previous similar message [ 4144.903879] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4153.733775] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4169.841360] Lustre: *** cfs_fail_loc=159, val=0*** [ 4185.518713] Lustre: lustre-MDT0000: Client 5173cf27-6a8d-476e-b359-0053c25b1223 (at 192.168.202.1@tcp) reconnecting [ 4187.204042] Lustre: *** cfs_fail_loc=159, val=0*** [ 4187.211201] Lustre: Skipped 5 previous similar messages [ 4203.414248] Lustre: lustre-MDT0000: Client 5173cf27-6a8d-476e-b359-0053c25b1223 (at 192.168.202.1@tcp) reconnecting [ 4210.145446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4210.157090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.175569] Lustre: Skipped 1 previous similar message [ 4210.181943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4210.195441] Lustre: Skipped 1 previous similar message [ 4213.224635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4216.316947] Lustre: server umount lustre-OST0000 complete [ 4221.409702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4221.413314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.429135] Lustre: Skipped 1 previous similar message [ 4221.433983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4221.436858] Lustre: Skipped 2 previous similar messages [ 4226.436504] Lustre: server umount lustre-MDT0000 complete [ 4228.577467] LustreError: 131038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4228.601044] LustreError: 131038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4229.845845] LustreError: 131017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441500 with bad export cookie 2210287416384553708 [ 4229.866242] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4230.374584] Lustre: server umount lustre-MDT0001 complete [ 4243.289909] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 4246.757364] Key type lgssc unregistered [ 4247.126865] LNet: 134835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4247.147859] LNetError: 134835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4248.165797] LNet: Removed LNI 192.168.202.101@tcp [ 4249.044388] Key type .llcrypt unregistered [ 4249.048389] Key type ._llcrypt unregistered [ 4277.006393] Key type ._llcrypt registered [ 4277.008729] Key type .llcrypt registered [ 4277.129208] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 4278.480703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4278.751185] alg: No test for adler32 (adler32-zlib) [ 4279.959338] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4280.301668] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 4282.033242] Key type lgssc registered [ 4283.413562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4292.375898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4292.399246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4293.923684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.174372] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4303.335455] LustreError: 136250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4303.361555] LustreError: 136250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4308.454046] LustreError: 136249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4310.928974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.508529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4317.500297] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4326.524817] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4333.119444] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4340.417515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4340.810436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.660963] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4356.533207] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4363.384266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4376.475396] Lustre: *** cfs_fail_loc=159, val=0*** [ 4392.846895] Lustre: lustre-MDT0000: Client f49b277b-5164-482d-a0d4-2c9c1c472beb (at 192.168.202.1@tcp) reconnecting [ 4395.709770] Lustre: *** cfs_fail_loc=159, val=0*** [ 4411.294059] Lustre: lustre-MDT0001: Client f49b277b-5164-482d-a0d4-2c9c1c472beb (at 192.168.202.1@tcp) reconnecting [ 4413.273198] Lustre: *** cfs_fail_loc=159, val=0*** [ 4413.273198] Lustre: *** cfs_fail_loc=159, val=0*** [ 4413.273207] Lustre: Skipped 3 previous similar messages [ 4413.293281] Lustre: Skipped 1 previous similar message [ 4429.717783] Lustre: lustre-MDT0001: Client f49b277b-5164-482d-a0d4-2c9c1c472beb (at 192.168.202.1@tcp) reconnecting [ 4435.712650] Lustre: 139309:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4439.051190] Lustre: *** cfs_fail_loc=159, val=0*** [ 4439.057770] Lustre: Skipped 2 previous similar messages [ 4455.296456] Lustre: lustre-MDT0000: Client 8294d843-6baa-424f-8c9a-9cf705a7fff9 (at 192.168.202.1@tcp) reconnecting [ 4457.881799] Lustre: *** cfs_fail_loc=159, val=0*** [ 4457.884171] Lustre: Skipped 15 previous similar messages [ 4473.245517] Lustre: lustre-MDT0000: Client 8294d843-6baa-424f-8c9a-9cf705a7fff9 (at 192.168.202.1@tcp) reconnecting [ 4475.227284] Lustre: 139601:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4481.504930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4481.510677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4481.540278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4484.578646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4484.584876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4484.593290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4486.650183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4486.666330] Lustre: Skipped 1 previous similar message [ 4487.444697] Lustre: server umount lustre-OST0000 complete [ 4494.443728] Lustre: server umount lustre-MDT0000 complete [ 4495.846222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4495.852980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4495.889630] LustreError: 136249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.560983] LustreError: 136231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441770 with bad export cookie 12470110494986009854 [ 4499.577179] LustreError: 136231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4499.578768] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4500.164964] Lustre: server umount lustre-MDT0001 complete [ 4512.715897] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 4516.023629] Key type lgssc unregistered [ 4516.476802] LNet: 140652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4516.481853] LNetError: 140652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4516.504546] LNet: Removed LNI 192.168.202.101@tcp [ 4517.418444] Key type .llcrypt unregistered [ 4517.432333] Key type ._llcrypt unregistered [ 4547.125858] Key type ._llcrypt registered [ 4547.131784] Key type .llcrypt registered [ 4547.277952] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 4548.951876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4549.017867] alg: No test for adler32 (adler32-zlib) [ 4550.165884] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4550.444109] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 4552.143330] Key type lgssc registered [ 4553.491967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4561.820296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4561.867931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.514436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.980934] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4573.664686] LustreError: 142065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.705572] LustreError: 142065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4578.796598] LustreError: 142064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.673970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4580.096120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4585.310523] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4595.488443] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4602.986904] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4613.145570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.919266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.459657] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4625.242190] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4632.920495] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4633.443240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4638.819902] Lustre: 144648:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4643.315957] Lustre: 144697:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4645.420649] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4646.423658] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4650.992693] Lustre: 144843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4652.291925] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4653.274632] Lustre: 144941:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4659.178145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4659.179054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.196507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4659.216664] Lustre: Skipped 1 previous similar message [ 4659.233514] Lustre: Skipped 1 previous similar message [ 4664.339355] Lustre: server umount lustre-OST0000 complete [ 4669.414079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.433607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4669.443455] Lustre: Skipped 1 previous similar message [ 4674.527869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4674.538031] Lustre: Skipped 2 previous similar messages [ 4674.818530] Lustre: server umount lustre-MDT0000 complete [ 4678.914104] LustreError: 142044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441949 with bad export cookie 2115542925186963744 [ 4678.915349] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4678.930639] LustreError: 142044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4679.437631] Lustre: server umount lustre-MDT0001 complete [ 4694.352697] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 4698.308929] Key type lgssc unregistered [ 4698.697374] LNet: 146003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4698.724834] LNetError: 146003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4699.753695] LNet: Removed LNI 192.168.202.101@tcp [ 4701.052382] Key type .llcrypt unregistered [ 4701.055960] Key type ._llcrypt unregistered [ 4732.086539] Key type ._llcrypt registered [ 4732.088367] Key type .llcrypt registered [ 4732.232291] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 4733.357516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4733.496876] alg: No test for adler32 (adler32-zlib) [ 4734.588183] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4734.835296] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 4736.535749] Key type lgssc registered [ 4737.774398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4746.339659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4746.376087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4747.941394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.860696] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4757.473915] LustreError: 147417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4757.505557] LustreError: 147417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4761.603868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.164227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4766.787968] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4774.562948] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4780.388293] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4788.138907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4788.555864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.645463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4796.027673] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4804.864627] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4806.977962] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4814.076251] Lustre: *** cfs_fail_loc=159, val=0*** [ 4830.630677] Lustre: lustre-MDT0000: Client d7f883de-855c-4c3f-9c4c-93fa7d539c94 (at 192.168.202.1@tcp) reconnecting [ 4837.856740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4837.868252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.872708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4837.890204] Lustre: Skipped 1 previous similar message [ 4837.907991] Lustre: Skipped 1 previous similar message [ 4842.439288] Lustre: server umount lustre-OST0000 complete [ 4848.096744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4848.107126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4848.112222] Lustre: Skipped 1 previous similar message [ 4848.144680] Lustre: Skipped 1 previous similar message [ 4853.217169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4853.225523] Lustre: Skipped 1 previous similar message [ 4857.319727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.335395] Lustre: Skipped 1 previous similar message [ 4860.383320] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4860.605643] Lustre: server umount lustre-MDT0000 complete [ 4862.441363] LustreError: 149267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.465916] LustreError: 149267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4864.943528] LustreError: 147397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442135 with bad export cookie 15337324603767060298 [ 4864.961579] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.975929] LustreError: 147397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4865.061337] LustreError: 146727:0:(import.c:761:ptlrpc_connect_import_locked()) already connecting [ 4865.585576] Lustre: server umount lustre-MDT0001 complete [ 4878.695825] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 4882.009501] Key type lgssc unregistered [ 4882.372417] LNet: 151081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4882.388817] LNetError: 151081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4882.410850] LNet: Removed LNI 192.168.202.101@tcp [ 4883.499740] Key type .llcrypt unregistered [ 4883.503070] Key type ._llcrypt unregistered [ 4909.350059] Key type ._llcrypt registered [ 4909.351756] Key type .llcrypt registered [ 4909.507431] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 4910.468993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4910.556074] alg: No test for adler32 (adler32-zlib) [ 4911.651480] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4911.928279] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 4913.640217] Key type lgssc registered [ 4915.010972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4923.755643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4923.793364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4925.491811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4930.797421] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4935.657143] LustreError: 152495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4935.686801] LustreError: 152495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4940.611675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4940.768263] LustreError: 152496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.232370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4945.785864] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4954.690313] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4961.493807] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4968.450410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4968.917104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.034015] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4975.331782] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 4983.854272] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4985.399461] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 5003.950827] Lustre: lustre-OST0000: evicting 6d5f90d7-29b4-4582-8767-cfe19e3b4f15 (at 192.168.202.1@tcp) by administrative request [ 5003.955741] Lustre: Skipped 1 previous similar message [ 5036.515113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5036.519039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.556244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5039.675601] Lustre: server umount lustre-OST0000 complete [ 5043.681255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5043.689534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.707601] Lustre: Skipped 1 previous similar message [ 5043.714464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5043.728527] Lustre: Skipped 1 previous similar message [ 5046.752883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.755295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.858753] Lustre: server umount lustre-MDT0000 complete [ 5051.876607] LustreError: 152491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5051.898280] LustreError: 152491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5054.342704] LustreError: 152478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442325 with bad export cookie 16749858142005991273 [ 5054.361715] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5054.370960] LustreError: 152478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5054.662754] Lustre: server umount lustre-MDT0001 complete [ 5065.865802] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 5068.694531] Key type lgssc unregistered [ 5068.998682] LNet: 156566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5069.005922] LNetError: 156566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5070.060111] LNet: Removed LNI 192.168.202.101@tcp [ 5070.985476] Key type .llcrypt unregistered [ 5070.990318] Key type ._llcrypt unregistered [ 5156.358470] Key type ._llcrypt registered [ 5156.359915] Key type .llcrypt registered [ 5156.464164] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 5158.347072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5158.491613] alg: No test for adler32 (adler32-zlib) [ 5159.605227] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5159.969595] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 5161.655336] Key type lgssc registered [ 5163.453504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5173.301702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5173.323076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5174.910747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5180.157353] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 5184.486528] LustreError: 159059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5184.509426] LustreError: 159059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5189.600768] LustreError: 159060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5190.760444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.287318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5196.267339] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 5204.637505] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5211.143744] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5219.606813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5220.014214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5225.991442] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 5234.715925] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5235.393325] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5244.596514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5255.139409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5255.142992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.163341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5259.149944] Lustre: server umount lustre-OST0000 complete [ 5265.377487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5265.386067] Lustre: Skipped 1 previous similar message [ 5265.390802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5265.393722] Lustre: Skipped 1 previous similar message [ 5269.271414] Lustre: server umount lustre-MDT0000 complete [ 5270.496790] LustreError: 160262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5270.522162] LustreError: 160262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5273.064873] LustreError: 159040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442543 with bad export cookie 7429892821289877680 [ 5273.075112] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5273.079292] LustreError: 159040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5273.541281] Lustre: server umount lustre-MDT0001 complete [ 5284.134327] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 5286.861020] Key type lgssc unregistered [ 5287.070726] LNet: 162625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5287.076069] LNetError: 162625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5287.090852] LNet: Removed LNI 192.168.202.101@tcp [ 5287.900314] Key type .llcrypt unregistered [ 5287.903432] Key type ._llcrypt unregistered [ 5313.607497] Key type ._llcrypt registered [ 5313.609802] Key type .llcrypt registered [ 5313.715324] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 5314.764849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5314.882241] alg: No test for adler32 (adler32-zlib) [ 5315.972923] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5316.239825] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 5317.887229] Key type lgssc registered [ 5319.092751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5335.622405] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 09:36:44 (1788442604) [ 5352.107149] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 5362.913351] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 5402.786208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5402.811360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.162524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5404.233205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5404.343519] Lustre: lustre-MDT0000: new disk, initializing [ 5404.479501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.501968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5409.288660] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 5421.193385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.414357] Lustre: 166889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5421.542384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5421.547268] Lustre: Skipped 1 previous similar message [ 5421.778618] Lustre: lustre-MDT0001: new disk, initializing [ 5421.962038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5422.065515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5422.088683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5428.964557] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 5441.266206] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5447.954980] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5455.858296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5456.064728] Lustre: lustre-OST0000: new disk, initializing [ 5456.067872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5456.073403] Lustre: 168854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5456.158505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5463.999370] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 5465.668249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5465.682204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5465.781366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5473.975707] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5486.565195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.570526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5486.595508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5486.601607] LustreError: Skipped 1 previous similar message [ 5486.625227] Lustre: Skipped 1 previous similar message [ 5491.353679] Lustre: server umount lustre-OST0000 complete [ 5496.803390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5496.805804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5496.819211] Lustre: Skipped 2 previous similar messages [ 5501.939715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5501.949214] Lustre: Skipped 2 previous similar messages [ 5502.357979] Lustre: server umount lustre-MDT0000 complete [ 5506.032511] LustreError: 166880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442776 with bad export cookie 17078822803346130089 [ 5506.033486] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5506.048641] LustreError: 166880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5506.735852] Lustre: server umount lustre-MDT0001 complete [ 5518.213667] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 5521.221995] Key type lgssc unregistered [ 5521.476915] LNet: 170760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5521.482215] LNetError: 170760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5521.494773] LNet: Removed LNI 192.168.202.101@tcp [ 5522.320896] Key type .llcrypt unregistered [ 5522.323175] Key type ._llcrypt unregistered [ 5546.755971] Key type ._llcrypt registered [ 5546.757290] Key type .llcrypt registered [ 5546.911422] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 5548.194441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5548.245599] alg: No test for adler32 (adler32-zlib) [ 5549.359905] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5549.577913] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 5551.233950] Key type lgssc registered [ 5552.518554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5560.886074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5560.912544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5562.743509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.281333] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 5573.098553] LustreError: 172175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.117619] LustreError: 172175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5576.278022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.672809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5581.271142] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 5588.666535] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5594.567762] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5601.726613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.149572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.292203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5609.520230] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 5617.285234] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5623.072402] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5623.120065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5623.154477] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5623.162769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.181238] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5624.091416] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5624.122287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.138679] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5624.144448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.155307] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5625.002986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.824875] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5625.833407] Lustre: Skipped 1 previous similar message [ 5625.842737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.851755] Lustre: Skipped 1 previous similar message [ 5625.855381] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5625.859859] Lustre: Skipped 1 previous similar message [ 5625.867319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.889416] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5625.892161] Lustre: Skipped 1 previous similar message [ 5626.793277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.613967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.502413] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5628.511155] Lustre: Skipped 2 previous similar messages [ 5628.532052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.540513] Lustre: Skipped 2 previous similar messages [ 5628.544865] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5628.552974] Lustre: Skipped 2 previous similar messages [ 5628.556914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.567046] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5628.576845] Lustre: Skipped 2 previous similar messages [ 5629.396449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.206602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.103173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.226095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.345146] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5633.359241] Lustre: Skipped 4 previous similar messages [ 5633.385644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.398075] Lustre: Skipped 4 previous similar messages [ 5633.402415] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5633.405965] Lustre: Skipped 4 previous similar messages [ 5633.409301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.416797] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5633.423293] Lustre: Skipped 4 previous similar messages [ 5634.506330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.372726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.215675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.252598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.207354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.132371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.192194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.179287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.379874] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5642.384828] Lustre: Skipped 8 previous similar messages [ 5642.397987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.408778] Lustre: Skipped 8 previous similar messages [ 5642.413716] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5642.419207] Lustre: Skipped 8 previous similar messages [ 5642.423849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.440690] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5642.450282] Lustre: Skipped 8 previous similar messages [ 5643.385071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.453670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.630663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.795526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.801932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.100338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.312864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.234853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.195608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.123755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.384490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.420813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.372480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.615986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.546654] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5658.549660] Lustre: Skipped 14 previous similar messages [ 5658.561323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.572660] Lustre: Skipped 14 previous similar messages [ 5658.581172] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5658.586860] Lustre: Skipped 14 previous similar messages [ 5658.596633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.609467] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5658.622118] Lustre: Skipped 14 previous similar messages [ 5659.558453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.451485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.579848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.486093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.352158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.183626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.999606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.900539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.983787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.927665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.802897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.649452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.572680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.500250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.470871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.287898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.287955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.315288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.532724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.806961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.842442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.164907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.283866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.242704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.462388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.659736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.004946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.043318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.957502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.708608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.546401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.323775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.368742] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5691.379283] Lustre: Skipped 32 previous similar messages [ 5691.399471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.408368] Lustre: Skipped 32 previous similar messages [ 5691.412641] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5691.418564] Lustre: Skipped 32 previous similar messages [ 5691.421677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.427412] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5691.430189] Lustre: Skipped 32 previous similar messages [ 5692.372616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.406720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.440303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.682891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.820042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.921773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.243279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.290209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.380550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.337408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.535130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.570536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.447858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.395896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.224056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.283438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.385349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.440511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.380965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.413291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.586501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.340486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.258913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.466042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.095154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.077500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.892790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.895069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.856885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.832405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.649206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.542987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.354176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.202568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.007887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.830853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.600718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.286801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.014512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.777499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.534284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.428241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.296288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.302900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.216137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.184293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.051302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.191894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.175525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.275905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.136489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.179479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.191479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.377492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.408742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.443770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.440459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.413663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.614987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.824976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.166368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.562142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.735701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.740161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.724387] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5755.736234] Lustre: Skipped 64 previous similar messages [ 5755.769603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.790738] Lustre: Skipped 64 previous similar messages [ 5755.797584] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5755.809752] Lustre: Skipped 64 previous similar messages [ 5755.815682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.834620] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5755.845060] Lustre: Skipped 64 previous similar messages [ 5756.843161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.059883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.735760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.012550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.361987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.709379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.664134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.943962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.355915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.718942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.984687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.008401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.282634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.414968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.328488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.498303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.575257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.712700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.771943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.039073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.468738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.550680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.625048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.866178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.065991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.220277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.302326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.229485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.239063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.079732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.108950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.823574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.911265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.775232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.874267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.824334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.817752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.719354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.798887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.789495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.997069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.957866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.010472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.017482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.985121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.957201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.293263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.345812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.634523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.742574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.173890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.331956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.787852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.045299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.296960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.169482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.329198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.586242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.359333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.489774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.487641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.431519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.377131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.392618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.293317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.181716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.226977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.340147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.244507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.238484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.189261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.191155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.207694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.133755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.949801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.948862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.856509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.802091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.753641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.914503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.989392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.873806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.002741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.150556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.328780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.244525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.404538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.572969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.801328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.818783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.064753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.127878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.297735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.002434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.252932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.396830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.558389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.721378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.722896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.949176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.143843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.178119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.497297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.886065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.025462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.963213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.152601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.086867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.162586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.311503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.291508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.451450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.448378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.581039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.674786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.785642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.131805] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5884.145590] Lustre: Skipped 116 previous similar messages [ 5884.170945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5884.205373] Lustre: Skipped 116 previous similar messages [ 5884.234480] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5884.244592] Lustre: Skipped 116 previous similar messages [ 5884.252984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.265898] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5884.276388] Lustre: Skipped 116 previous similar messages [ 5885.298622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.356154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.340253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.725244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.879331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.834086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.897559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.064140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.795923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.637976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.531116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.548764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.456847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.662670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.553401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.516098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.257804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.182054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.043489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.205721] LustreError: 172195:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5904.211892] LustreError: 172195:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5904.225233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.220101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.103688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.273279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.197223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.221706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.680943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.127803] LustreError: 172195:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5912.158178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.163712] LustreError: 172195:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5913.375995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.712546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.243520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.608778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.639408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.052735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.279062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.689273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.025639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.293141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.936202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.363219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.826484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.298436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.268553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.349676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.607184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.157372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.314383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.065284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.346849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.440244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.573511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.428080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.601725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.603102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.551618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.518831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.316812] LustreError: 172195:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5947.321352] LustreError: 172195:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5947.332059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.161530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.155496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.000785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.893237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.429794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.979381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.671403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.644143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.547529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.391415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.355472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.241186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.034258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.204049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.209136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.323669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.667165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.713637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.664140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.590182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.493835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.324985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.285117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.512437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.681063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.679035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.833625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.856165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.923969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.123031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.288785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.368061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.433044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.798238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.077903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.142784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.256435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.485205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.382049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.580507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.740391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.848130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.926189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.364916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.669383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.055277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.386301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.334749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.623116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.776675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.969166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.898910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.867725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.713321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.875792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.891625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.914125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.806787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.888821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.820248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.817257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.602748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.639656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.591932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.483379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.390966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.491242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.493865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.614279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.559768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.494403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.302695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.269358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.155948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.058929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.305055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.381720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.603359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.781452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.818115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.080236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.359682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.643179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.883802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.066453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.530231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.949152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.212035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.702891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.822831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.793123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.835598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.511839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.461886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.533582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.713540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.816358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.649711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.544795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.663397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.644605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.512842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.655387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.516508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.627126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.539575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.688290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.615769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.914814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.226047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.325878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.401136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.633050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.835157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.274773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.537626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.560515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.844192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.901968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.638290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.852320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.765935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.015629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.237920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.163159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.160958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.960839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.964865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.891677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.809328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.882070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.908956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.823164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.976043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.934460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.009820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.895795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.003825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.990804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.920565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.879886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.030829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.112654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.051743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.185402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.104841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.066188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.017070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.138610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.937928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.935536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.952041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.031855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.099419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.166971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.362547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.754720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.847657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.246492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.447346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.971627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.369856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.433607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.808681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.253918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.803553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.976942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.220689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.364894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.680719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.655607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.789778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.741310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.804858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.746949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.736539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.523770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.524379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.393541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.451470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.312522] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6140.318169] Lustre: Skipped 236 previous similar messages [ 6140.330352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6140.359918] Lustre: Skipped 236 previous similar messages [ 6140.374655] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6140.387110] Lustre: Skipped 236 previous similar messages [ 6140.393703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.403532] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6140.422289] Lustre: Skipped 236 previous similar messages [ 6141.544704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.415072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.515930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.669271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.652561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.705911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.938419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.268601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.205250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.170527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.087234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.883524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.849606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.136170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.359362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.476458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.504502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.747318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.610703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.621795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.697785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.905097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.271108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.242172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.361751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.594604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.475203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.647057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.743173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.859679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.921292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.926370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.038236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.099508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.256309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.436176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.558209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.727335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.745009] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.009193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.306919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.405303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.435117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.449131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.445958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.422648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.411382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.433817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.417460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.567962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.475285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.571060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.669539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.625714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.584387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.452847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.494681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.430943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.339844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.321425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.482139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.496702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.583727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.839448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.778466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.864892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.801529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.827856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.772646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.678360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.612512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.551699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.512360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.382903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.811531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.751711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.097984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.320305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.216668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.282906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.150694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.923699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.809731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.841923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.962203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.099386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.115843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.248552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.482579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.391641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.649334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.802063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.830837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.897496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.097881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.360840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.382726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.602031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.546069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6246.890604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.313345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.442684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.486346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.462326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.452230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.375391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.263371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.312449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.292606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.477815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.512174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.378370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.332217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.353143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.255965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.026551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.889371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.879833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.910659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.789957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.611869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.462225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.337326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.381834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.433422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.578942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.457838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.212916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.216181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.144942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.086386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.005360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.769725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.635274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.372928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.391796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.226604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.382188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.471355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.696425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.910250] LustreError: 172195:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6286.917231] LustreError: 172195:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6286.925229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.029036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.303512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.282730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.561597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.618634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.930082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.857428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.246217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.171181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.447953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.743779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.105614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.459918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.637669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.573418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.973766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.215140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.343203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.772743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.785621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.406300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.639365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.025062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.401328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.433775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.683606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.423704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.409509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.476527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.675260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.673481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.641425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.623224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.486225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.398418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.439142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.523788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.412511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.424539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.319571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.284264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.224056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.254772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.259407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.275061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.173230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.119765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.008794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.854639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.768924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.066472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.031144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.901032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.830607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.681397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.664285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.997194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.920501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.833950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.726489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.738554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.129361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.666146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.937445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.165647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.453602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.915712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.075495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.159457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.587248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.968375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.098984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.347509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.323219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.790897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.812539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.971938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.853975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.934514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.778456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.047864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.999630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.794475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.641988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.603165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.456636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.319039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.194272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.286273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.207535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.220321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.202655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.176371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.089749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.122709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.898304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.766317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.572836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.670392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.548811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.558115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.575739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.462194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.415064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.491720] LustreError: 172195:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6399.500511] LustreError: 172195:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6399.517935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.447995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.440359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.354500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.526629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.447858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.561443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.453414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.425227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.197283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.277313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.240705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.200358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.107093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.156959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.489968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.822822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.088527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.965770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.034881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.106865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.041058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.886599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.669971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.452318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.329783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.439876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.511529] LustreError: 172195:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6426.537890] LustreError: 172195:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6426.544157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.565205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.537364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.411962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.522089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.641968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.651353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.592984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.465691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.332274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.280030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.300525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.959899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.766124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.928737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.924509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.947850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.984596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.409077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.481468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.370487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.391370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.499617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.869773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.031582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.265322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.874408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.212088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.925074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.287793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.686582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.892571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.876399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.696137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.980938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.650302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.221439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.567630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.239524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.372821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.850469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.869074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.453484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.362817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.407769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.190724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.320962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.898406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.830123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.995533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.073223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.977209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.226668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.499062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.536049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.495651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.312669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.346712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.005876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.003399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.986535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.212805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.195659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.339433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.275090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.306623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.253362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.402739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.542538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.522018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.457956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.326982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.249795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.224548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.179585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.983152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.039116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.024473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.460476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.682405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.845594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.830805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.996657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.086742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.110419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.288048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.498635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.481437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.632836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.614562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.865927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.970786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.972532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.007242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.968755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.956957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.897668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.124877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.081770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.012496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.175301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.326417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.271664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6543.237397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.384514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.337971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.257773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.363402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.309591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.426152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.723205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.960287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.012290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.557560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.665103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.780620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.050540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.821088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.701053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.752451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.862187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.056937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.055565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6565.342742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.477133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.546505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.761510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6570.085717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6571.523608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6573.117053] LustreError: 172195:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6573.131641] LustreError: 172195:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6573.144936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.687704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6575.529666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.586668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6578.291775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6579.602739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.029171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.301903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.285859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6584.488851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6585.814692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.488221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6588.321518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.678089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6591.089303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.230245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.391729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6594.415292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6595.490825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6596.466940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.307593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.396739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6599.126087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6599.960237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6600.994219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6601.757572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6602.914849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6604.211912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6605.342927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6606.305811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6607.509554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6608.613459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.901991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6611.057213] LustreError: 172195:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6611.072561] LustreError: 172195:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6611.100595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6612.414525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.398476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6614.340092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6615.401104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6616.458060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6617.672352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6618.743750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6619.959971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6621.067261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6621.993626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.169841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6624.307231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6625.130153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6626.205871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6627.157024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6628.431979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6629.222526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6630.207249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6631.021814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6632.250728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6633.116754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6634.360035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6635.192975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6636.427576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6637.244343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6638.176794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6639.171077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6640.345440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6641.530737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6642.285863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6643.351491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6644.465197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6645.859900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6647.443937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6649.113260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6650.470664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6651.908840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6653.332723] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6653.341271] Lustre: Skipped 472 previous similar messages [ 6653.361986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6653.371690] Lustre: Skipped 472 previous similar messages [ 6653.375978] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6653.379315] Lustre: Skipped 472 previous similar messages [ 6653.385714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6653.401655] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6653.414541] Lustre: Skipped 472 previous similar messages [ 6654.874306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6656.223822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6657.050712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6658.255031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6659.769287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6661.320550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6663.045057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6664.435217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6665.730883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6667.159917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6668.233854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6669.515943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6670.579120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6671.798195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6673.416560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6674.753146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6675.989727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6677.313364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6679.131640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6680.357274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6681.294233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6682.401839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6683.513105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6685.001500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6686.379598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6687.542602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6688.982422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6690.211476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6691.398814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6692.358254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6693.560402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6694.606336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6695.616444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6696.905846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6697.860372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6698.718496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6699.745620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6700.547321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6701.520261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7134.517557] LustreError: 173998:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 7134.526613] LustreError: 172195:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7810.026298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7810.029959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.038177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7810.052834] Lustre: Skipped 40 previous similar messages [ 7810.071609] Lustre: Skipped 1 previous similar message [ 7811.806582] Lustre: server umount lustre-OST0000 complete [ 7819.232914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7819.242102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7820.259506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7820.269054] Lustre: Skipped 1 previous similar message [ 7821.676478] Lustre: server umount lustre-MDT0000 complete [ 7825.383207] LustreError: 172176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7825.412231] LustreError: 172176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7825.558285] LustreError: 172156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445096 with bad export cookie 1362873616354069773 [ 7825.559252] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7825.566222] LustreError: 172156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7826.319410] Lustre: server umount lustre-MDT0001 complete [ 7839.913288] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 7843.208639] Key type lgssc unregistered [ 7843.542535] LNet: 227464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7843.550490] LNetError: 227464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7843.581155] LNet: Removed LNI 192.168.202.101@tcp [ 7844.547340] Key type .llcrypt unregistered [ 7844.549481] Key type ._llcrypt unregistered [ 7877.369173] Key type ._llcrypt registered [ 7877.371146] Key type .llcrypt registered [ 7877.465989] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 7879.145443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7879.194938] alg: No test for adler32 (adler32-zlib) [ 7880.468916] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7880.979564] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 7882.823215] Key type lgssc registered [ 7884.373275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7892.383859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7892.423052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.938169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.833591] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 7903.205126] LustreError: 228881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.227502] LustreError: 228881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7908.328411] LustreError: 228881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7908.616219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.881517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7913.350420] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 7922.005993] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7928.621659] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7936.765296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7937.056427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7939.138803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7943.580406] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 7951.832641] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7957.514879] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7959.229340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.266168] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7959.292329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7959.319513] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7959.319641] LustreError: 228901:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7963.629278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7963.640501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.656858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7964.645801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.653219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7964.685197] Lustre: Skipped 1 previous similar message [ 7967.755401] Lustre: server umount lustre-OST0000 complete [ 7979.351173] LustreError: 228901:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 8005.236683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8005.776790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.863371] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8014.945200] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8014.958359] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8020.990217] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8029.290393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8029.304849] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8034.013939] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 9 sec [ 8040.830456] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f7fc309b000.ost_server_uuid 50 [ 8042.903497] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f7fc309b000.ost_server_uuid in FULL state after 0 sec [ 8048.096364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8048.102713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8048.119952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8048.122507] Lustre: Skipped 1 previous similar message [ 8050.155079] LustreError: 232089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.184526] LustreError: 232089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8050.319805] Lustre: server umount lustre-OST0000 complete [ 8055.269754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.273548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8055.297220] Lustre: Skipped 2 previous similar messages [ 8055.310329] Lustre: Skipped 1 previous similar message [ 8060.011689] Lustre: server umount lustre-MDT0000 complete [ 8060.391596] LustreError: 228877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8060.438189] LustreError: 228877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8064.632172] LustreError: 230723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445335 with bad export cookie 1082756115838271023 [ 8064.637247] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8064.639287] LustreError: 230723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8064.898971] Lustre: server umount lustre-MDT0001 complete [ 8076.726535] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 8079.923982] Key type lgssc unregistered [ 8080.214265] LNet: 234110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8080.224420] LNetError: 234110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8080.257649] LNet: Removed LNI 192.168.202.101@tcp [ 8081.496519] Key type .llcrypt unregistered [ 8081.499093] Key type ._llcrypt unregistered [ 8121.508584] Key type ._llcrypt registered [ 8121.513628] Key type .llcrypt registered [ 8121.663710] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 8122.809368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8122.819383] alg: No test for adler32 (adler32-zlib) [ 8124.020915] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8124.237538] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 8125.879174] Key type lgssc registered [ 8126.853734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8136.983536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8137.009449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8138.547074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8142.910404] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8148.966699] LustreError: 235836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8148.987653] LustreError: 235836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8151.506157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8151.917596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8156.899262] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8160.068920] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8168.801471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8169.227938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8178.394229] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8183.549061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8191.994620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8192.237546] Lustre: lustre-OST0001: new disk, initializing [ 8192.241444] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8192.249338] Lustre: 238146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8192.339262] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8199.590466] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8199.802500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8199.816347] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8199.947461] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8211.796389] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8219.692820] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8232.493232] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8232.514074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8232.563565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8236.010475] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.012704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8236.054458] Lustre: Skipped 2 previous similar messages [ 8236.071152] Lustre: Skipped 3 previous similar messages [ 8239.243378] Lustre: server umount lustre-MDT0000 complete [ 8241.161448] LustreError: 235833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8241.187290] LustreError: 235833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8245.049638] LustreError: 235817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445515 with bad export cookie 6777770001015120951 [ 8245.071930] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8245.502462] Lustre: server umount lustre-MDT0001 complete [ 8261.032723] Lustre: server umount lustre-OST0000 complete [ 8262.879467] Lustre: 235097:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445517/real 1788445517] req@ffff944d04054000 x1875320918609024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8262.917851] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8266.246712] Lustre: server umount lustre-OST0001 complete [ 8289.279962] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 8299.956397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.381235] LustreError: 241257:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8300.482766] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8305.637035] LustreError: 241258:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8305.651595] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8310.755726] LustreError: 241257:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.171831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.635926] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8319.499856] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8322.496910] Lustre: 242398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8330.835398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8331.527161] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8338.899845] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8339.775887] LustreError: 242753:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.807792] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:129) [ 8346.104830] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8348.553264] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8356.160730] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8365.418372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8369.338897] Lustre: 244271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8385.004871] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8385.014700] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8385.024186] Lustre: Skipped 2 previous similar messages [ 8385.060605] Lustre: Skipped 3 previous similar messages [ 8388.353932] Lustre: server umount mylustre-MDT0000 complete [ 8390.117190] LustreError: 241257:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8390.143228] LustreError: 241257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8392.154354] LustreError: 241234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445662 with bad export cookie 6777770001015123814 [ 8392.163096] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8392.168082] LustreError: 241234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8392.700317] Lustre: server umount mylustre-MDT0001 complete [ 8407.794597] Lustre: server umount mylustre-OST0000 complete [ 8422.104115] Lustre: server umount mylustre-OST0001 complete [ 8446.345169] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 8456.715042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8457.205354] LustreError: 246581:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8457.308777] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8457.319345] Lustre: Skipped 1 previous similar message [ 8461.999753] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8471.996437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.318691] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8480.466537] Lustre: 247722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8486.626816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8492.770906] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8498.226993] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:161) [ 8501.299613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8502.520145] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8502.526661] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8507.332882] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8516.331841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8522.074953] Lustre: 249596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8533.483069] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8533.508635] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8533.514341] Lustre: Skipped 4 previous similar messages [ 8533.540171] Lustre: Skipped 3 previous similar messages [ 8536.307411] Lustre: server umount tfs-MDT0000 complete [ 8538.604803] LustreError: 247314:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8538.618827] LustreError: 247314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8542.202842] LustreError: 248904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445813 with bad export cookie 6777770001015125760 [ 8542.215606] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8542.405386] Lustre: server umount tfs-MDT0001 complete [ 8557.046575] Lustre: server umount tfs-OST0000 complete [ 8559.914744] Lustre: 235097:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445814/real 1788445814] req@ffff944d152c1c00 x1875320918728192/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8559.948390] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8560.966441] Lustre: server umount tfs-OST0001 complete [ 8583.581369] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 8593.710738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8594.006712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.009119] Lustre: Skipped 3 previous similar messages [ 8597.880737] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8606.655196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8611.981681] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8614.817527] Lustre: 252990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8621.745366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8627.939527] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8631.307154] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:100 to 0x280000400:193) [ 8636.869388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8639.192233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8639.195200] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8644.985176] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8653.971529] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8657.738189] Lustre: 254861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8666.227409] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:32:14 (1788445934) [ 8684.001090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8684.017354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8684.028695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8689.518885] Lustre: server umount lustre-MDT0000 complete [ 8690.660446] LustreError: 251846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8690.679752] LustreError: 251846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 8692.839100] LustreError: 251821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445963 with bad export cookie 6777770001015127685 [ 8692.842229] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8692.856551] LustreError: 251821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8695.784054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8695.795945] Lustre: Skipped 5 previous similar messages [ 8699.354665] Lustre: server umount lustre-MDT0001 complete [ 8718.303266] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8718.715403] Lustre: server umount lustre-OST0000 complete [ 8730.880851] Lustre: server umount lustre-OST0001 complete [ 8737.889673] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 8748.268288] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 8785.197438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.413491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8785.451741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8785.537116] Lustre: lustre-MDT0000: new disk, initializing [ 8785.651654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8785.656298] Lustre: Skipped 3 previous similar messages [ 8785.670356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8790.387728] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8801.501903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8801.641840] Lustre: 258051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8801.702705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8801.706626] Lustre: Skipped 1 previous similar message [ 8801.795920] Lustre: lustre-MDT0001: new disk, initializing [ 8801.982334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8802.000692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8806.042778] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8818.019756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8818.515580] Lustre: lustre-OST0000: new disk, initializing [ 8818.524393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8818.533805] Lustre: 259632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8820.237498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8820.243833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8820.378357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8825.640146] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8838.730921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8838.959237] Lustre: lustre-OST0001: new disk, initializing [ 8838.961285] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8838.964547] Lustre: 260505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8840.868896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8840.891945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8840.992494] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8848.639992] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 8862.974262] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 10:35:31 (1788446131) [ 8881.193302] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 10:35:49 (1788446149) [ 8896.999867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8897.020749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8897.042374] Lustre: Skipped 5 previous similar messages [ 8897.064418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8897.076679] Lustre: Skipped 2 previous similar messages [ 8910.303154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8910.608443] Lustre: server umount lustre-OST0000 complete [ 8921.595297] Lustre: server umount lustre-MDT0000 complete [ 8925.287207] LustreError: 258041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446196 with bad export cookie 6777770001015129463 [ 8925.297430] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8925.300022] LustreError: 258041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8930.272613] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8930.279279] Lustre: Skipped 10 previous similar messages [ 8931.922571] Lustre: server umount lustre-MDT0001 complete [ 8944.710424] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 9019.048231] Lustre: server umount lustre-OST0001 complete [ 9026.217268] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 9042.806897] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 9083.729798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9084.087443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9084.126909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9084.226413] Lustre: lustre-MDT0000: new disk, initializing [ 9084.412641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.425033] Lustre: Skipped 3 previous similar messages [ 9084.485310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9090.550992] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9101.887095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.987928] Lustre: 265360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9102.041991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9102.048518] Lustre: Skipped 1 previous similar message [ 9102.161854] Lustre: lustre-MDT0001: new disk, initializing [ 9102.306833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9102.324058] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9108.028507] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9119.736095] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9125.426363] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9132.904688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9133.305491] Lustre: 267329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9134.907155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9134.917406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9135.031610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9140.235910] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9150.796823] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9160.677198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9160.685413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9160.693018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9160.701248] LustreError: Skipped 2 previous similar messages [ 9160.735572] Lustre: Skipped 6 previous similar messages [ 9164.175206] Lustre: server umount lustre-OST0000 complete [ 9168.865083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9174.574855] Lustre: server umount lustre-MDT0000 complete [ 9176.038940] LustreError: 265367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.075917] LustreError: 265367:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 9178.479180] LustreError: 267328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446449 with bad export cookie 6777770001015130933 [ 9178.486162] LustreError: 267328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9178.488621] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9178.798331] Lustre: server umount lustre-MDT0001 complete [ 9193.161189] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 9196.851679] Key type lgssc unregistered [ 9197.226713] LNet: 269091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9197.238695] LNetError: 269091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9197.295487] LNet: Removed LNI 192.168.202.101@tcp [ 9198.241145] Key type .llcrypt unregistered [ 9198.244807] Key type ._llcrypt unregistered [ 9229.123822] Key type ._llcrypt registered [ 9229.126495] Key type .llcrypt registered [ 9229.236913] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 9231.063891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9231.107606] alg: No test for adler32 (adler32-zlib) [ 9232.284536] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9232.548734] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 9234.199952] Key type lgssc registered [ 9235.174598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9243.551655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9243.579945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9244.958668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9249.815893] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9254.371248] LustreError: 270551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9254.384246] LustreError: 270551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9258.867719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.188536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9263.798982] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9271.725795] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9277.737783] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9285.368606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9285.713255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.880098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9292.335651] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9300.731120] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9301.768856] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9305.573221] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9305.583934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9305.601326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9308.669967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.680588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9308.682637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9309.788461] Lustre: server umount lustre-OST0000 complete [ 9313.760158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9313.763791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9313.771597] Lustre: Skipped 1 previous similar message [ 9313.789302] Lustre: Skipped 2 previous similar messages [ 9318.886909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9318.894809] Lustre: Skipped 1 previous similar message [ 9319.853461] Lustre: server umount lustre-MDT0000 complete [ 9323.772338] LustreError: 273391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446594 with bad export cookie 12944422320462255762 [ 9323.773504] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9323.780775] LustreError: 273391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9324.179436] Lustre: server umount lustre-MDT0001 complete [ 9339.134916] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 9342.547454] Key type lgssc unregistered [ 9342.833803] LNet: 274142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9342.844429] LNetError: 274142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9342.863068] LNet: Removed LNI 192.168.202.101@tcp [ 9343.744092] Key type .llcrypt unregistered [ 9343.745889] Key type ._llcrypt unregistered [ 9362.423467] Key type ._llcrypt registered [ 9362.425381] Key type .llcrypt registered [ 9362.519586] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 9363.810579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9363.952607] alg: No test for adler32 (adler32-zlib) [ 9364.981468] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9365.189536] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 9366.872666] Key type lgssc registered [ 9367.880824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9376.878384] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9376.890866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9392.415452] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9397.535423] LustreError: 275332:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.101@tcp: failed processing log, type 4: rc = -110 [ 9424.223256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9430.542615] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9438.918264] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9441.650870] Lustre: server umount lustre-OST0000 complete [ 9460.676235] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [ 9463.301369] Key type lgssc unregistered [ 9463.563662] LNet: 276963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9463.568966] LNetError: 276963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9464.627655] LNet: Removed LNI 192.168.202.101@tcp [ 9465.803682] Key type .llcrypt unregistered [ 9465.809901] Key type ._llcrypt unregistered [ 9506.024797] Key type ._llcrypt registered [ 9506.027252] Key type .llcrypt registered [ 9506.120341] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 9507.126114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9507.156676] alg: No test for adler32 (adler32-zlib) [ 9508.206729] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9508.419804] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [ 9510.071142] Key type lgssc registered [ 9511.404548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9525.167865] loop0: detected capacity change from 0 to 204800000 [ 9525.420608] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9525.450844] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9525.907924] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9526.987471] loop0: detected capacity change from 0 to 204800000 [ 9527.244193] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9527.262653] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9527.637704] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9528.801772] loop0: detected capacity change from 0 to 204800000 [ 9528.977135] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9528.994729] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9529.099707] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9530.167591] loop0: detected capacity change from 0 to 204800000 [ 9530.403589] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9530.512301] print_req_error: 1 callbacks suppressed [ 9530.512312] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9531.657662] loop0: detected capacity change from 0 to 204800000 [ 9531.701046] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9535.758809] loop0: detected capacity change from 0 to 204800000 [ 9535.798977] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9539.908371] loop0: detected capacity change from 0 to 204800000 [ 9539.952669] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9544.667855] loop0: detected capacity change from 0 to 204800000 [ 9544.701348] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9549.693660] loop0: detected capacity change from 0 to 204800000 [ 9549.789081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9549.872637] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9549.985834] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9550.007977] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9550.023564] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9550.039716] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9550.049862] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9550.064594] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9550.074416] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9550.082546] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9550.092752] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9550.100562] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9550.108831] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9550.125644] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9550.139734] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9550.147215] Lustre: 280375:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9556.185219] Lustre: server umount MGS complete [ 9557.253573] loop0: detected capacity change from 0 to 204800000 [ 9557.402965] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9558.012687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9558.086954] Lustre: lustre-MDT0000: Aborting client recovery [ 9559.301391] loop1: detected capacity change from 0 to 204800000 [ 9559.425942] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9559.582973] Lustre: 280808:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9559.593728] Lustre: 280808:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9559.613263] Lustre: 280808:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9559.626207] Lustre: 280808:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9559.648892] Lustre: 280808:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9559.668785] Lustre: 280808:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9559.687330] Lustre: 280808:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9559.737674] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9560.086801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9560.181454] Lustre: lustre-MDT0001: Aborting client recovery [ 9561.421281] loop2: detected capacity change from 0 to 204800000 [ 9561.504194] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9561.577790] Lustre: 280886:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9561.590126] Lustre: 280886:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9561.597708] Lustre: 280886:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9561.606065] Lustre: 280886:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9561.620125] Lustre: 280886:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9561.628582] Lustre: 280886:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9561.636490] Lustre: 280886:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9561.763069] Lustre: Found index 0 for lustre-OST0000, updating log [ 9561.788238] Lustre: 280904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9561.864926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9561.899823] Lustre: lustre-OST0000: Aborting recovery [ 9562.772068] loop3: detected capacity change from 0 to 204800000 [ 9562.961569] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9563.036022] Lustre: 280969:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9563.055543] Lustre: 280969:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9563.073462] Lustre: 280969:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9563.086681] Lustre: 280969:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9563.103681] Lustre: 280969:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9563.123796] Lustre: 280969:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9563.138257] Lustre: 280969:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9563.176480] Lustre: Found index 1 for lustre-OST0001, updating log [ 9563.218213] Lustre: 280974:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9563.302297] Lustre: lustre-OST0001: Aborting recovery [ 9567.399719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9567.410056] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9567.423280] LustreError: 280752:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9567.440364] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9567.447559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9569.470953] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9569.483361] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9572.757869] LustreError: 280718:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1345us: rc = -2 [ 9573.258932] LustreError: 280716:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 739us: rc = -2 [ 9573.304278] LustreError: 280716:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 20 previous similar messages [ 9590.240924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.242869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9590.256406] Lustre: Skipped 2 previous similar messages [ 9590.260914] Lustre: Skipped 1 previous similar message [ 9592.774646] Lustre: server umount lustre-MDT0000 complete [ 9593.665117] LustreError: 280700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446864 with bad export cookie 15490448751692535781 [ 9593.690821] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9595.360219] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9595.361381] LustreError: 280716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9595.374959] Lustre: Skipped 1 previous similar message [ 9595.376361] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9595.376367] Lustre: Skipped 2 previous similar messages [ 9595.444124] LustreError: 280716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9598.368953] LustreError: 280982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9598.369737] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9598.381252] LustreError: 280982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9598.387309] Lustre: Skipped 2 previous similar messages [ 9600.118067] Lustre: server umount lustre-MDT0001 complete [ 9615.329404] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9615.606358] Lustre: server umount lustre-OST0000 complete [ 9631.199979] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9631.514653] Lustre: server umount lustre-OST0001 complete [ 9640.562687] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 10:48:28 (1788446908) [ 9668.699421] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [ 9679.627780] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [ 9714.677591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9714.860692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9714.879635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9714.969617] Lustre: lustre-MDT0000: new disk, initializing [ 9715.051967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9715.065541] Lustre: Skipped 1 previous similar message [ 9715.084410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9719.167456] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9728.369958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9728.499089] Lustre: 285044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9728.577865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9728.585211] Lustre: Skipped 1 previous similar message [ 9728.636300] Lustre: lustre-MDT0001: new disk, initializing [ 9728.716160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9728.751408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9728.766504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9732.657913] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9742.535347] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9747.820374] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9754.926358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9755.371628] Lustre: lustre-OST0000: new disk, initializing [ 9755.381063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9755.390451] Lustre: 287011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9755.463164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9762.854078] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9764.909790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9764.930783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9765.090143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9774.342847] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9780.175960] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9791.824981] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9799.861530] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9847.264325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9847.284704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9847.297465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9847.306447] Lustre: Skipped 2 previous similar messages [ 9847.343663] Lustre: Skipped 1 previous similar message [ 9850.716290] Lustre: server umount lustre-OST0000 complete [ 9856.999222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9857.032366] LustreError: Skipped 1 previous similar message [ 9857.054476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9857.091268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9861.281055] Lustre: server umount lustre-MDT0000 complete [ 9862.629936] LustreError: 285054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9862.651540] LustreError: 285054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9865.589315] LustreError: 287868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447136 with bad export cookie 15490448751692538504 [ 9865.597308] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9865.599138] LustreError: 287868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9866.036593] Lustre: server umount lustre-MDT0001 complete [ 9874.669123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.375094] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9884.215468] Lustre: server umount MGS complete [ 9896.160094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9896.563626] LustreError: 290926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.584819] LustreError: 290939:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9896.637414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.079155] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9902.054203] LustreError: 290927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9910.433825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9915.602314] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9924.075731] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9932.167931] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9940.667256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9941.214188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9941.216665] Lustre: Skipped 1 previous similar message [ 9947.537610] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [ 9955.209898] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9986.528982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9986.548278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9986.557879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9986.557892] Lustre: Skipped 2 previous similar messages [ 9986.570609] Lustre: Skipped 2 previous similar messages [ 9992.517758] Lustre: server umount lustre-OST0000 complete [ 9996.772724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10001.892623] LustreError: 292771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10001.905458] LustreError: 292771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10001.962209] Lustre: server umount lustre-MDT0000 complete [10005.977058] LustreError: 290908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447276 with bad export cookie 15490448751692540324 [10005.987016] LustreError: 290908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10005.989053] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10006.543834] Lustre: server umount lustre-MDT0001 complete [10017.235179] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [10019.747810] Key type lgssc unregistered [10020.044898] LNet: 294874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10020.053301] LNetError: 294874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10020.078515] LNet: Removed LNI 192.168.202.101@tcp [10020.721752] Key type .llcrypt unregistered [10020.724167] Key type ._llcrypt unregistered [10055.281621] Key type ._llcrypt registered [10055.292126] Key type .llcrypt registered [10055.378900] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [10069.643807] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [10070.676888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10070.758628] alg: No test for adler32 (adler32-zlib) [10071.872938] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10072.059746] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [10073.696075] Key type lgssc registered [10075.087932] Lustre: Echo OBD driver; http://www.lustre.org/ [10115.645855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10115.679698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10117.084143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10117.156044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10117.275824] Lustre: lustre-MDT0000: new disk, initializing [10117.349758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10117.374383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10120.745145] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [10131.953957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10132.089807] Lustre: 298828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10132.148214] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10132.154366] Lustre: Skipped 1 previous similar message [10132.277474] Lustre: lustre-MDT0001: new disk, initializing [10132.445462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10132.548586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10132.565783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10137.371294] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [10149.602674] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10157.313955] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10164.335910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.667642] Lustre: lustre-OST0000: new disk, initializing [10164.671164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10164.682218] Lustre: 300795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10164.782852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10171.950339] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [10173.030568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10173.046082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10173.174471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10182.345722] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10190.214886] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10203.679213] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10207.641897] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10255.334885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10255.342702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10255.345104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10255.353214] Lustre: Skipped 1 previous similar message [10257.689516] Lustre: server umount lustre-OST0000 complete [10265.577163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10265.578061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10265.579035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10265.579040] Lustre: Skipped 1 previous similar message [10265.592955] LustreError: Skipped 1 previous similar message [10265.633410] Lustre: Skipped 1 previous similar message [10268.009932] Lustre: server umount lustre-MDT0000 complete [10272.484942] LustreError: 298821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447543 with bad export cookie 7447556924230339824 [10272.487862] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10272.498701] LustreError: 298821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10272.927045] Lustre: server umount lustre-MDT0001 complete [10286.154648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.855843] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [10296.955226] Lustre: server umount MGS complete [10309.116771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10309.619455] LustreError: 304660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.642229] LustreError: 304660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10309.656117] LustreError: 304673:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10309.723717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10314.594944] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [10314.735340] LustreError: 304661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.843260] LustreError: 304660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10323.531174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10323.992734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10329.569928] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [10340.484276] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10348.007295] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10357.733576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10358.367729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10366.082635] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [10375.247926] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10412.003478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10412.011559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10412.015656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10412.015666] Lustre: Skipped 1 previous similar message [10412.063450] Lustre: Skipped 1 previous similar message [10415.015401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10415.027377] Lustre: Skipped 2 previous similar messages [10420.204181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10425.312965] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10425.505365] Lustre: server umount lustre-OST0000 complete [10430.432408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.440882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10430.441869] Lustre: Skipped 1 previous similar message [10430.457460] Lustre: Skipped 4 previous similar messages [10436.359277] Lustre: server umount lustre-MDT0000 complete [10440.445618] LustreError: 306507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447711 with bad export cookie 7447556924230341672 [10440.453387] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10440.465422] LustreError: 306507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10440.856409] Lustre: server umount lustre-MDT0001 complete [10453.915647] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [10457.636746] Key type lgssc unregistered [10458.089107] LNet: 308621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10458.100606] LNetError: 308621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10458.124721] LNet: Removed LNI 192.168.202.101@tcp [10459.167386] Key type .llcrypt unregistered [10459.169716] Key type ._llcrypt unregistered [10481.651213] Key type ._llcrypt registered [10481.652555] Key type .llcrypt registered [10481.774865] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [10497.046743] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [10498.100338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10498.134682] alg: No test for adler32 (adler32-zlib) [10499.385285] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10499.691356] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [10501.423213] Key type lgssc registered [10502.897872] Lustre: Echo OBD driver; http://www.lustre.org/ [10530.696591] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [10546.182419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10546.233889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.528736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10547.571557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10547.674821] Lustre: lustre-MDT0000: new disk, initializing [10547.780523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10547.803809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10552.862952] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [10565.979741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10566.317348] Lustre: lustre-OST0000: new disk, initializing [10566.330799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10566.338171] Lustre: Skipped 1 previous similar message [10566.354267] Lustre: 312669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10566.567949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10574.007216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10574.035113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10574.108411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10577.793441] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11474.105034] Lustre: server umount lustre-MDT0000 complete [11478.085871] LustreError: 311857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448748 with bad export cookie 6125237920426274876 [11478.102406] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11487.876459] Lustre: server umount lustre-OST0000 complete [11515.215784] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [11525.673416] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [11569.009390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11569.406900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11569.467988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11569.591332] Lustre: lustre-MDT0000: new disk, initializing [11569.672466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11569.696475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11575.116497] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11586.152868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11586.289906] Lustre: 317014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11586.381513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11586.387525] Lustre: Skipped 1 previous similar message [11586.428248] Lustre: lustre-MDT0001: new disk, initializing [11586.474584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11586.507602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11586.519818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11591.510304] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11602.887584] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11610.574023] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11618.118335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11618.565615] Lustre: lustre-OST0000: new disk, initializing [11618.569328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11618.574588] Lustre: 318984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11618.707700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11624.371609] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11627.049716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11627.064661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11627.173216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11634.643331] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11639.511586] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11639.688432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11644.803410] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11645.064280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11647.975919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11647.976588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11647.995330] LustreError: Skipped 1 previous similar message [11647.998258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11648.020182] Lustre: Skipped 1 previous similar message [11648.034839] Lustre: Skipped 1 previous similar message [11653.095631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11653.100688] Lustre: Skipped 1 previous similar message [11653.899781] Lustre: server umount lustre-OST0000 complete [11663.337070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11663.351245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11663.360921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11663.385922] Lustre: Skipped 1 previous similar message [11664.850786] Lustre: server umount lustre-MDT0000 complete [11668.932915] LustreError: 317808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448939 with bad export cookie 6125237920429215695 [11668.937511] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11668.958087] LustreError: 317808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11669.320844] Lustre: server umount lustre-MDT0001 complete [11673.237676] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11675.020415] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 11:22:23 (1788448943) [11682.574521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.112331] LustreError: 321296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11683.139854] LustreError: 321296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11683.203683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11688.102693] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11688.424966] LustreError: 321297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11692.528929] LustreError: 321296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11696.128876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11696.421408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11700.519482] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11707.515687] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11712.707330] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11719.031574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11719.446283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11725.636614] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11732.181407] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11738.366401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11738.503205] Lustre: lustre-OST0001: new disk, initializing [11738.506915] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11738.513330] Lustre: 324131:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11738.594579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11744.776333] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11745.379119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11745.389050] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11745.444273] LustreError: 324515:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11746.468917] LustreError: 324515:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11746.488495] LustreError: 324515:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11747.568110] LustreError: 324515:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11747.586149] LustreError: 324515:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11749.674146] LustreError: 324524:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11749.691845] LustreError: 324524:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [11753.495085] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11753.668824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11753.765105] LustreError: 324515:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11753.773368] LustreError: 324515:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [11759.175632] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9f7fd2528000.ost_server_uuid 50 [11761.020608] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9f7fd2528000.ost_server_uuid in FULL state after 0 sec [11761.957684] LustreError: 324515:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11761.975212] LustreError: 324515:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11768.030799] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f7fd2528000.ost_server_uuid 50 [11769.769106] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f7fd2528000.ost_server_uuid in FULL state after 0 sec [11771.276125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11785.610140] Lustre: lustre-OST0001: Not available for connect from 192.168.202.1@tcp (stopping) [11785.619466] Lustre: Skipped 1 previous similar message [11786.721030] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11786.722706] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.733156] LustreError: Skipped 1 previous similar message [11786.775638] Lustre: Skipped 1 previous similar message [11789.795505] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11789.805632] Lustre: Skipped 3 previous similar messages [11791.540814] Lustre: server umount lustre-OST0001 complete [11794.923518] LustreError: 323118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11794.963657] LustreError: 323118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11798.504828] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11798.514807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11798.526923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11798.535392] Lustre: Skipped 1 previous similar message [11800.051642] LustreError: 323118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11800.065724] LustreError: 323118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11803.741465] Lustre: server umount lustre-OST0000 complete [11809.250961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11809.261117] LustreError: Skipped 1 previous similar message [11809.269289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11809.286157] Lustre: Skipped 1 previous similar message [11813.679923] Lustre: server umount lustre-MDT0000 complete [11815.393487] LustreError: 321293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11815.426672] LustreError: 321293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11818.863359] LustreError: 321276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449089 with bad export cookie 6125237920429216577 [11818.879523] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11819.491238] Lustre: server umount lustre-MDT0001 complete [11832.133126] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [11835.381971] Key type lgssc unregistered [11835.870892] LNet: 326819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11835.881650] LNetError: 326819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11836.901810] LNet: Removed LNI 192.168.202.101@tcp [11838.123366] Key type .llcrypt unregistered [11838.126957] Key type ._llcrypt unregistered [11874.841333] Key type ._llcrypt registered [11874.842735] Key type .llcrypt registered [11875.003912] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [11877.078278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11877.109521] alg: No test for adler32 (adler32-zlib) [11878.480187] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11878.890468] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [11880.591910] Key type lgssc registered [11882.265743] Lustre: Echo OBD driver; http://www.lustre.org/ [11890.725984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11890.753816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11892.449420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11897.623676] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11902.952287] LustreError: 328240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.970367] LustreError: 328240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11905.781587] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11908.067863] LustreError: 328241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11912.639201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11913.141122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11917.782497] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11926.765308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11927.064700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11933.278504] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11937.342299] LustreError: 329921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.363428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11941.197568] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11942.374891] LustreError: 329922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.402496] LustreError: 329922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11947.498945] LustreError: 329921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11947.519537] LustreError: 329921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11947.651232] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11947.892441] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11953.171233] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11954.170495] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [11962.552662] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11971.277551] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11971.518572] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12081.126150] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12081.129101] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12081.132156] LustreError: Skipped 1 previous similar message [12081.145914] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12081.152720] Lustre: Skipped 1 previous similar message [12081.188177] Lustre: Skipped 1 previous similar message [12082.829834] Lustre: server umount lustre-OST0001 complete [12086.249219] LustreError: 330451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12086.292231] LustreError: 330451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12090.849193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12090.852951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12090.858530] LustreError: Skipped 1 previous similar message [12090.859551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12090.894776] Lustre: Skipped 1 previous similar message [12096.548083] Lustre: server umount lustre-OST0000 complete [12101.602995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12101.621120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12101.627591] Lustre: Skipped 1 previous similar message [12101.650065] Lustre: Skipped 4 previous similar messages [12106.724779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12106.745828] Lustre: Skipped 1 previous similar message [12107.564897] Lustre: server umount lustre-MDT0000 complete [12111.681061] LustreError: 328222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449382 with bad export cookie 18354623482233369552 [12111.687570] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12111.688479] LustreError: 328222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12111.963440] Lustre: server umount lustre-MDT0001 complete [12124.570646] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [12127.861248] Key type lgssc unregistered [12128.192743] LNet: 333184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12128.201588] LNetError: 333184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12128.228457] LNet: Removed LNI 192.168.202.101@tcp [12129.349826] Key type .llcrypt unregistered [12129.351449] Key type ._llcrypt unregistered [12182.787342] Key type ._llcrypt registered [12182.790975] Key type .llcrypt registered [12182.977142] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [12184.175260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12184.465142] alg: No test for adler32 (adler32-zlib) [12185.693931] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12185.940949] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [12187.631371] Key type lgssc registered [12188.790088] Lustre: Echo OBD driver; http://www.lustre.org/ [12198.737741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12198.758133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12200.261113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12204.992864] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12210.660985] LustreError: 335474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12210.697965] LustreError: 335474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12215.776530] LustreError: 335473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12216.277341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12216.775819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12222.257563] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12225.256455] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12233.285339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12233.732530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12241.044506] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12241.900418] LustreError: 336960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12245.994330] LustreError: 336963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12246.023201] LustreError: 336963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12246.039612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12250.649471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12250.846255] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12256.266357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12257.116832] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12267.597233] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12275.812933] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12292.074654] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12292.084569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12292.117669] Lustre: Skipped 2 previous similar messages [12292.123733] Lustre: Skipped 1 previous similar message [12297.187896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12297.204513] Lustre: Skipped 5 previous similar messages [12298.160633] Lustre: server umount lustre-MDT0000 complete [12302.308390] LustreError: 335469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12302.341536] LustreError: 335469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12302.530293] LustreError: 335455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449573 with bad export cookie 15194003789790261593 [12302.530959] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12302.542253] LustreError: 335455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12303.126307] Lustre: server umount lustre-MDT0001 complete [12318.126645] Lustre: server umount lustre-OST0000 complete [12333.011255] Lustre: server umount lustre-OST0001 complete [12352.251211] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [12365.507034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12366.056988] LustreError: 340741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12366.161530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12371.217286] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12380.013244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12380.530805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12385.041734] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12388.065238] Lustre: 341881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12395.183133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12402.170781] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12405.927497] LustreError: 342237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12405.948242] LustreError: 342237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12411.070493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12412.551129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12412.871949] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12412.877885] Lustre: Skipped 1 previous similar message [12418.053984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12419.648484] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12427.858049] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12432.069658] Lustre: 343753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12438.499833] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.504829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12438.524652] Lustre: Skipped 3 previous similar messages [12438.545587] Lustre: Skipped 3 previous similar messages [12443.468078] Lustre: server umount lustre-MDT0000 complete [12443.617715] LustreError: 340742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12443.633214] LustreError: 340742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12447.060682] LustreError: 340722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449717 with bad export cookie 15194003789790263294 [12447.062488] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12447.072085] LustreError: 340722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12447.524377] Lustre: server umount lustre-MDT0001 complete [12462.066706] Lustre: server umount lustre-OST0000 complete [12465.126483] Lustre: 334737:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449719/real 1788449719] req@ffff944e416a6a00 x1875325176891520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788449735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12465.169625] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12465.188443] Lustre: Skipped 1 previous similar message [12466.139267] Lustre: server umount lustre-OST0001 complete [12469.708537] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12471.649676] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12473.520324] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 11:35:42 (1788449742) [12496.895510] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [12506.568472] loop0: detected capacity change from 0 to 1099511627776 [12507.230915] SGI XFS with ACLs, security attributes, quota, no debug enabled [12507.237124] *** VALIDATE xfs *** [12507.247629] XFS (loop0): Mounting V5 Filesystem [12507.276288] XFS (loop0): Ending clean mount [12508.868325] loop1: detected capacity change from 0 to 18691697672192 [12636.134120] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [12649.305365] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [12689.989686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12690.337600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12690.418774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12690.624825] Lustre: lustre-MDT0000: new disk, initializing [12690.774812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12690.793203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12696.416426] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12707.472826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12707.589530] Lustre: 348891:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12707.636747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12707.640783] Lustre: Skipped 1 previous similar message [12707.694683] Lustre: lustre-MDT0001: new disk, initializing [12707.806076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12707.812882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12713.171668] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12722.894444] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12729.581539] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12737.028300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12737.307342] Lustre: lustre-OST0000: new disk, initializing [12737.312964] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12737.322090] Lustre: 350861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12739.163866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12739.188835] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12739.283118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12744.391893] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12755.368796] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12759.873886] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12760.135296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12764.743083] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12764.992329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12770.283995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12770.287581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12770.306824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12770.309077] Lustre: Skipped 1 previous similar message [12770.321775] Lustre: Skipped 1 previous similar message [12775.399428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12781.023181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12781.181576] Lustre: server umount lustre-OST0000 complete [12785.637089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12785.650302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12785.661673] Lustre: Skipped 4 previous similar messages [12790.894768] Lustre: server umount lustre-MDT0000 complete [12794.531406] LustreError: 348884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450065 with bad export cookie 15194003789790264848 [12794.538355] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12794.550888] LustreError: 348884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12794.964523] Lustre: server umount lustre-MDT0001 complete [12802.874807] XFS (loop0): Unmounting Filesystem [12808.128296] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:41:16 (1788450076) [12814.708786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12814.996474] LustreError: 353424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12815.058286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12815.065929] Lustre: Skipped 2 previous similar messages [12818.681602] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12827.850320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12832.521977] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12840.114625] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12846.333466] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12852.841383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12859.657563] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12867.596568] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12879.843819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12879.850229] LustreError: Skipped 1 previous similar message [12879.858124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12879.882968] Lustre: Skipped 1 previous similar message [12879.888952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12879.893591] Lustre: Skipped 2 previous similar messages [12882.932627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12886.095534] Lustre: server umount lustre-OST0000 complete [12896.001765] Lustre: server umount lustre-MDT0000 complete [12899.789188] LustreError: 353404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450170 with bad export cookie 15194003789790265730 [12899.810199] LustreError: 353404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12899.824405] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12900.465407] Lustre: server umount lustre-MDT0001 complete [12914.265335] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [12917.091109] Key type lgssc unregistered [12917.362990] LNet: 357051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12917.369775] LNetError: 357051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12917.385569] LNet: Removed LNI 192.168.202.101@tcp [12918.415176] Key type .llcrypt unregistered [12918.417658] Key type ._llcrypt unregistered [12947.607620] Key type ._llcrypt registered [12947.614189] Key type .llcrypt registered [12947.729048] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [12948.794276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12949.150544] alg: No test for adler32 (adler32-zlib) [12950.259379] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12950.531816] LNet: Added LNI 192.168.202.101@tcp [8/256/0/180] [12952.184815] Key type lgssc registered [12953.115746] Lustre: Echo OBD driver; http://www.lustre.org/ [12959.632711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12959.656925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12961.114204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12965.167717] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12970.476274] LustreError: 358619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12970.485699] LustreError: 358619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12972.713271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12973.064134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12976.913336] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [12982.654573] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12987.272970] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12992.934844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12993.198513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12998.239261] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [13005.208391] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13012.416621] Lustre: server umount lustre-MDT0001 complete [13012.853377] LustreError: 360257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13013.986334] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13013.995054] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13014.004427] LustreError: 358618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13016.034554] LustreError: 359346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13016.040798] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13021.174076] LustreError: 358619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13021.190155] LustreError: 358619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13021.741377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13021.816323] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13021.820717] Lustre: Found index 1 for lustre-MDT0001, updating log [13021.859779] Lustre: 358607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13022.165445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13026.469492] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [13027.312753] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13029.367605] LustreError: 362214:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13029.378555] LustreError: 362214:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13029.390645] LustreError: 358611:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13029.503143] LustreError: 362217:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.101@tcp: cfg command failed: rc = -17 [13029.517583] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13029.517583] [13334.504137] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13334.522126] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13334.540396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13334.548195] Lustre: Skipped 1 previous similar message [13335.416525] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.1@tcp (stopping) [13337.925672] Lustre: server umount lustre-MDT0001 complete [13339.620062] LustreError: 358615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13339.646765] LustreError: 358615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13350.409443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13350.595409] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13350.602903] Lustre: Found index 1 for lustre-MDT0001, updating log [13350.693500] Lustre: 358608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13351.238292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13356.550827] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13357.647743] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [13360.652744] LustreError: 370698:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13360.670079] LustreError: 370698:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13360.691174] LustreError: 358611:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13360.701528] LustreError: 358611:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13360.789278] LustreError: 370705:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13360.796302] LustreError: 370705:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13360.801381] LustreError: 370705:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.101@tcp: cfg command failed: rc = -17 [13360.812893] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.202.101@tcp [13360.812893] [13668.235451] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.1@tcp (stopping) [13668.836042] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13668.836107] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13668.869164] Lustre: Skipped 2 previous similar messages [13670.949833] Lustre: server umount lustre-MDT0001 complete [13673.307700] LustreError: 358614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.1@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13673.329297] LustreError: 358614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [13681.416101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13681.526684] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13681.531577] Lustre: Found index 1 for lustre-MDT0001, updating log [13681.574831] Lustre: 358608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13681.915253] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13687.018267] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [13687.283854] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13690.362983] LustreError: 378962:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13690.383199] LustreError: 378962:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13690.405786] LustreError: 358611:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13690.430377] LustreError: 358611:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13690.518367] LustreError: 378965:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13690.538664] LustreError: 378965:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.101@tcp: cfg command failed: rc = -114 [13690.548512] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13690.548512] [14009.832576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14009.856223] Lustre: Skipped 2 previous similar messages [14009.868126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14009.880243] Lustre: Skipped 3 previous similar messages [14014.950233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14014.963863] Lustre: Skipped 2 previous similar messages [14024.159239] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14024.510848] Lustre: server umount lustre-MDT0000 complete [14025.189352] LustreError: 358613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14025.213707] LustreError: 358613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [14028.385707] LustreError: 358598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451299 with bad export cookie 5588744839908320348 [14028.394137] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14028.394520] LustreError: 358598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14030.305363] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14030.336772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14030.353801] Lustre: Skipped 3 previous similar messages [14034.908965] Lustre: server umount lustre-MDT0001 complete [14039.225270] Lustre: server umount lustre-OST0000 complete [14050.083218] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [14062.925555] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [14104.846992] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 12:02:53 (1788451373) [14113.209181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14113.549291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14113.574482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14113.695166] Lustre: lustre-MDT0000: new disk, initializing [14113.805684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14113.827938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14119.861552] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [14131.649073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14131.805589] Lustre: 389210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14131.851131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14131.854567] Lustre: Skipped 1 previous similar message [14131.912794] Lustre: lustre-MDT0001: new disk, initializing [14131.985994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14132.061072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14132.074098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14137.048329] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [14148.046573] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14156.490890] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14164.198706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14164.527783] Lustre: lustre-OST0000: new disk, initializing [14164.531732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14164.536259] Lustre: 391180:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14164.641170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14171.227156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14171.243633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14171.430403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14171.738488] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_default_debug -1 all [14182.163257] Lustre: DEBUG MARKER: oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14192.003908] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [14192.018370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14192.046306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14192.058550] Lustre: Skipped 1 previous similar message [14192.631850] Lustre: server umount lustre-MDT0000 complete [14196.509696] LustreError: 390005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451467 with bad export cookie 5588744839908322301 [14196.514847] LustreError: MGC192.168.202.101@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14196.519840] LustreError: 390005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14196.524249] LustreError: 390017:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [14196.889147] Lustre: server umount lustre-MDT0001 complete [14211.727687] Lustre: server umount lustre-OST0000 complete [14247.138313] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing set_hostid [14260.127828] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing load_modules_local [14299.680947] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14001 sec ========= 12:06:07 (1788451567) [14301.940819] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:06:10 (1788451570) === [14306.080825] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:06:14 (1788451574) === [14352.199745] Lustre: DEBUG MARKER: oleg201-server.virtnet: executing unload_modules_local [14355.106264] Key type lgssc unregistered [14355.459995] LNet: 397549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14355.464519] LNetError: 397549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14355.479354] LNet: Removed LNI 192.168.202.101@tcp [14356.444772] Key type .llcrypt unregistered [14356.449058] Key type ._llcrypt unregistered