[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 489757476 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003350] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.006006] kvm-guest: setup PV IPIs [ 0.008580] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010016] pid_max: default: 32768 minimum: 301 [ 0.012045] LSM: Security Framework initializing [ 0.013069] Yama: becoming mindful. [ 0.014057] SELinux: Initializing. [ 0.015072] *** VALIDATE selinux *** [ 0.023346] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027456] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030132] *** VALIDATE tmpfs *** [ 0.031510] *** VALIDATE proc *** [ 0.033191] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038039] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042014] debug: unmapping init [mem 0xffffffff90659000-0xffffffff90660fff] [ 0.045218] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046702] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047034] ... version: 2 [ 0.048020] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050020] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054361] rcu: Hierarchical SRCU implementation. [ 0.056640] smp: Bringing up secondary CPUs ... [ 0.057603] x86: Booting SMP configuration: [ 0.058034] .... node #0, CPUs: #1 #2 #3 [ 0.061477] smp: Brought up 1 node, 4 CPUs [ 0.063017] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.197023] node 0 deferred pages initialised in 130ms [ 0.200156] devtmpfs: initialized [ 0.201247] x86/mm: Memory block size: 128MB [ 0.203718] gcov: version magic: 0x41383552 [ 0.205006] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.208109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.211347] pinctrl core: initialized pinctrl subsystem [ 0.213162] [ 0.213971] ************************************************************* [ 0.217015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.220015] ** ** [ 0.223024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.225011] ** ** [ 0.228013] ** This means that this kernel is built to expose internal ** [ 0.231014] ** IOMMU data structures, which may compromise security on ** [ 0.233013] ** your system. ** [ 0.236014] ** ** [ 0.238011] ** If you see this message and you are not debugging the ** [ 0.241015] ** kernel, report this immediately to your vendor! ** [ 0.243012] ** ** [ 0.246013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249015] ************************************************************* [ 0.251725] NET: Registered protocol family 16 [ 0.253510] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.256057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.259062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.263110] cpuidle: using governor menu [ 0.264573] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.266552] PCI: Using configuration type 1 for base access [ 0.269127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278091] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.281043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.285079] cryptd: max_cpu_qlen set to 1000 [ 0.288257] ACPI: Added _OSI(Module Device) [ 0.290068] ACPI: Added _OSI(Processor Device) [ 0.291012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.293013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299310] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.305430] ACPI: Interpreter enabled [ 0.307063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.309014] ACPI: Using IOAPIC for interrupt routing [ 0.310095] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.314380] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.324843] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.327046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.330020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.334069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.339351] acpiphp: Slot [2] registered [ 0.341137] acpiphp: Slot [5] registered [ 0.342080] acpiphp: Slot [6] registered [ 0.345165] acpiphp: Slot [7] registered [ 0.346154] acpiphp: Slot [8] registered [ 0.348118] acpiphp: Slot [9] registered [ 0.350126] acpiphp: Slot [10] registered [ 0.352105] acpiphp: Slot [3] registered [ 0.353104] acpiphp: Slot [4] registered [ 0.355157] acpiphp: Slot [11] registered [ 0.357130] acpiphp: Slot [12] registered [ 0.359126] acpiphp: Slot [13] registered [ 0.360115] acpiphp: Slot [14] registered [ 0.361100] acpiphp: Slot [15] registered [ 0.363173] acpiphp: Slot [16] registered [ 0.364143] acpiphp: Slot [17] registered [ 0.365091] acpiphp: Slot [18] registered [ 0.366106] acpiphp: Slot [19] registered [ 0.367067] acpiphp: Slot [20] registered [ 0.368104] acpiphp: Slot [21] registered [ 0.370104] acpiphp: Slot [22] registered [ 0.372125] acpiphp: Slot [23] registered [ 0.374129] acpiphp: Slot [24] registered [ 0.375118] acpiphp: Slot [25] registered [ 0.376189] acpiphp: Slot [26] registered [ 0.378075] acpiphp: Slot [27] registered [ 0.379116] acpiphp: Slot [28] registered [ 0.381219] acpiphp: Slot [29] registered [ 0.383110] acpiphp: Slot [30] registered [ 0.384103] acpiphp: Slot [31] registered [ 0.386087] PCI host bridge to bus 0000:00 [ 0.387017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.389022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.392033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.394024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.396030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.399050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.401178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.404501] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.408284] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.419855] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.424011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.427022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.429023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.431038] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.435686] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.438864] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.441087] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.445894] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.451022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.464020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.470021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.477432] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.487018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.498023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.525021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.537232] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.546021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.553021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.571019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.581160] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.588018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.596020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.616026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.627171] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.634020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.641020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.659024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.671540] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.677018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.687021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.704021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.718444] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.725018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.731022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.748016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.759848] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.761340] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.763330] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.766398] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.769292] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.773141] iommu: Default domain type: Passthrough [ 0.775480] SCSI subsystem initialized [ 0.777181] ACPI: bus type USB registered [ 0.778113] usbcore: registered new interface driver usbfs [ 0.780086] usbcore: registered new interface driver hub [ 0.782115] usbcore: registered new device driver usb [ 0.783191] pps_core: LinuxPPS API ver. 1 registered [ 0.785024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.788091] PTP clock support registered [ 0.791096] EDAC MC: Ver: 3.0.0 [ 0.794170] PCI: Using ACPI for IRQ routing [ 0.796904] NetLabel: Initializing [ 0.798013] NetLabel: domain hash size = 128 [ 0.799012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.801120] NetLabel: unlabeled traffic allowed by default [ 0.804103] vgaarb: loaded [ 0.806224] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.807011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.813000] clocksource: Switched to clocksource kvm-clock [ 0.920001] VFS: Disk quotas dquot_6.6.0 [ 0.922124] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.924953] *** VALIDATE ramfs *** [ 0.926261] *** VALIDATE hugetlbfs *** [ 0.927881] pnp: PnP ACPI init [ 0.930523] pnp: PnP ACPI: found 6 devices [ 0.976424] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.980420] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.982710] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.985273] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.988063] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.990855] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.994157] NET: Registered protocol family 2 [ 0.996863] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.002189] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.006266] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.011667] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.015615] TCP: Hash tables configured (established 65536 bind 65536) [ 1.018910] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.022306] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.025482] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.029035] NET: Registered protocol family 1 [ 1.032457] RPC: Registered named UNIX socket transport module. [ 1.035049] RPC: Registered udp transport module. [ 1.037036] RPC: Registered tcp transport module. [ 1.039065] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.041899] NET: Registered protocol family 44 [ 1.044403] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.047397] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.050406] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.053134] PCI: CLS 0 bytes, default 64 [ 1.055166] Unpacking initramfs... [ 2.474636] debug: unmapping init [mem 0xffff89b8bcc54000-0xffff89b8bffbffff] [ 2.479267] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.481073] software IO TLB: mapped [mem 0x00000000a3600000-0x00000000a7600000] (64MB) [ 2.483791] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.993669] Initialise system trusted keyrings [ 2.995512] Key type blacklist registered [ 2.998479] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.007729] zbud: loaded [ 3.038439] *** VALIDATE nfs *** [ 3.039868] *** VALIDATE nfs4 *** [ 3.042136] pstore: using deflate compression [ 3.051438] Platform Keyring initialized [ 3.160620] NET: Registered protocol family 38 [ 3.162402] Key type asymmetric registered [ 3.163542] Asymmetric key parser 'x509' registered [ 3.165378] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.168765] io scheduler mq-deadline registered [ 3.170087] io scheduler kyber registered [ 3.171514] io scheduler bfq registered [ 3.173187] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.175782] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.178182] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.180654] ACPI: Power Button [PWRF] [ 3.185595] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.191862] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.205844] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.216520] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.230390] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.261054] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.287796] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.292625] Non-volatile memory driver v1.3 [ 3.294207] Linux agpgart interface v0.103 [ 3.324559] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.326985] vda: detected capacity change from 0 to 76677120 [ 3.340850] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.343551] vdb: detected capacity change from 0 to 1073741824 [ 3.357597] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.360283] vdc: detected capacity change from 0 to 2621440000 [ 3.376088] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.378628] vdd: detected capacity change from 0 to 2621440000 [ 3.391670] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.394097] vde: detected capacity change from 0 to 4294967296 [ 3.407581] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.410466] vdf: detected capacity change from 0 to 4294967296 [ 3.421944] libphy: Fixed MDIO Bus: probed [ 3.427305] usbcore: registered new interface driver usbserial_generic [ 3.429606] usbserial: USB Serial support registered for generic [ 3.432112] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.437491] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.439722] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.442255] mousedev: PS/2 mouse device common for all mice [ 3.445743] rtc_cmos 00:05: RTC can wake from S4 [ 3.447772] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.449452] rtc_cmos 00:05: registered as rtc0 [ 3.451123] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.453674] intel_pstate: CPU model not supported [ 3.456097] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.459244] hid: raw HID events driver (C) Jiri Kosina [ 3.461026] usbcore: registered new interface driver usbhid [ 3.462781] usbhid: USB HID core driver [ 3.464183] drop_monitor: Initializing network drop monitor service [ 3.466386] Initializing XFRM netlink socket [ 3.466593] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.467982] NET: Registered protocol family 10 [ 3.471272] Segment Routing with IPv6 [ 3.472442] NET: Registered protocol family 17 [ 3.474100] mpls_gso: MPLS GSO support [ 3.479795] RAS: Correctable Errors collector initialized. [ 3.481658] AVX version of gcm_enc/dec engaged. [ 3.483215] AES CTR mode by8 optimization enabled [ 3.554522] sched_clock: Marking stable (3554493950, 0)->(4427330826, -872836876) [ 3.557595] registered taskstats version 1 [ 3.559544] Loading compiled-in X.509 certificates [ 3.561165] zswap: loaded using pool lzo/zbud [ 3.587330] Key type big_key registered [ 3.599460] Key type encrypted registered [ 3.600896] ima: No TPM chip found, activating TPM-bypass! [ 3.602751] ima: Allocated hash algorithm: sha1 [ 3.604531] ima: No architecture policies found [ 3.605906] evm: Initialising EVM extended attributes: [ 3.607614] evm: security.selinux [ 3.608586] evm: security.ima [ 3.609410] evm: security.capability [ 3.610333] evm: HMAC attrs: 0x1 [ 3.613558] rtc_cmos 00:05: setting system clock to 2026-09-03 12:24:00 UTC (1788438240) [ 3.619327] debug: unmapping init [mem 0xffffffff91603000-0xffffffff917fffff] [ 3.621745] debug: unmapping init [mem 0xffffffff90382000-0xffffffff90658fff] [ 3.630107] Write protecting the kernel read-only data: 28672k [ 3.632915] debug: unmapping init [mem 0xffffffff8ea03000-0xffffffff8ebfffff] [ 3.635245] debug: unmapping init [mem 0xffffffff8f314000-0xffffffff8f3fffff] [ 3.666585] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.672954] systemd[1]: Detected virtualization kvm. [ 3.674629] systemd[1]: Detected architecture x86-64. [ 3.676100] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.701559] systemd[1]: No hostname configured. [ 3.702922] systemd[1]: Set hostname to . [ 3.704497] random: systemd: uninitialized urandom read (16 bytes read) [ 3.706452] systemd[1]: Initializing machine ID from random generator. [ 3.747596] random: ln: uninitialized urandom read (6 bytes read) [ 3.837620] random: systemd: uninitialized urandom read (16 bytes read) [ 3.840681] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.845392] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.850604] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.480641] device-mapper: uevent: version 1.0.3 [ 4.482436] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System.[ 5.091911] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.223690] virtio_net virtio0 ens2: renamed from eth0 [ 5.343302] scsi host0: ata_piix [ 5.353643] scsi host1: ata_piix [ 5.355865] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.358749] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.946328] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.123927] random: crng init done [ 10.126134] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.550887] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.670251] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.920300] SELinux: Disabled at runtime. [ 11.984151] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.993400] systemd[1]: Detected virtualization kvm. [ 11.994938] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.528317] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.531741] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.537418] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.541929] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.545722] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.553258] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.557974] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached targe[ 12.665251] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS t Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.012727] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.351505] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.366974] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.466504] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.482526] EDAC sbridge: Ver: 1.1.2 [ 15.315932] Key type dns_resolver registered [ 15.621900] NFS: Registering the id_resolver key type [ 15.624064] Key type id_resolver registered [ 15.625875] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg643-server login: [ 68.179162] libcfs: loading out-of-tree module taints kernel. [ 68.223460] Key type ._llcrypt registered [ 68.225728] Key type .llcrypt registered [ 68.340900] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 85.444430] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 86.774554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.783252] alg: No test for adler32 (adler32-zlib) [ 88.179266] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 88.908183] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 90.671185] Key type lgssc registered [ 92.357929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 109.199357] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 119.252282] hrtimer: interrupt took 5105954 ns [ 151.062175] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 164.291336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 164.354889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.572217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.594214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.670582] Lustre: lustre-MDT0000: new disk, initializing [ 165.744627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.760864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.803380] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.998657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.091816] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.121582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.126554] Lustre: Skipped 1 previous similar message [ 182.208302] Lustre: lustre-MDT0001: new disk, initializing [ 182.260459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.293409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.305077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.531495] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.270318] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 200.369345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.648386] Lustre: lustre-OST0000: new disk, initializing [ 200.653301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 200.664668] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.757842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.769953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.785820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.875815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.238560] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.457529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.566291] Lustre: lustre-OST0001: new disk, initializing [ 219.569158] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.573699] Lustre: 9527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.621453] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.557883] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.934743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.949619] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 230.028804] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.688897] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.025957] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 255.638758] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing check_logdir /tmp/testlogs/ [ 261.138140] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing yml_node [ 265.375888] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 268.553526] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 270.980182] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 272.590770] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:28:27 EDT 2026 [ 289.672948] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 291.090753] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.659804] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.672497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.682880] Lustre: Skipped 3 previous similar messages [ 311.780442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.791298] Lustre: Skipped 3 previous similar messages [ 312.322144] Lustre: server umount lustre-MDT0000 complete [ 315.292661] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438552 with bad export cookie 291171525781775883 [ 315.294487] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.306868] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 316.900627] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.910400] Lustre: Skipped 3 previous similar messages [ 316.927704] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.936415] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.953696] Lustre: Skipped 2 previous similar messages [ 321.757856] Lustre: server umount lustre-MDT0001 complete [ 325.413364] Lustre: server umount lustre-OST0000 complete [ 328.817591] Lustre: server umount lustre-OST0001 complete [ 335.524718] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 344.802968] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 378.365199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.718547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 378.755044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 378.861938] Lustre: lustre-MDT0000: new disk, initializing [ 378.964111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 378.980425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 384.168449] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 395.620032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.740410] Lustre: 15075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 395.834305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 395.839949] Lustre: Skipped 1 previous similar message [ 395.923361] Lustre: lustre-MDT0001: new disk, initializing [ 396.024789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.064601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.075513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 399.846568] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 411.534289] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 417.013306] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 423.411092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.693647] Lustre: lustre-OST0000: new disk, initializing [ 423.697400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 423.707219] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 423.863094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.015814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 425.044712] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 425.228666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 430.511388] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 439.259594] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 443.684451] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 443.994724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.637813] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.807243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 451.046205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.049842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.060836] LustreError: Skipped 1 previous similar message [ 451.064382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.078647] Lustre: Skipped 2 previous similar messages [ 451.101073] Lustre: Skipped 1 previous similar message [ 456.894696] Lustre: server umount lustre-OST0000 complete [ 461.281887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.291361] Lustre: Skipped 1 previous similar message [ 461.293623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 461.300139] Lustre: Skipped 3 previous similar messages [ 475.104646] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 475.282503] Lustre: server umount lustre-MDT0000 complete [ 476.641554] LustreError: 15080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.654635] LustreError: 15080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 478.638724] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438715 with bad export cookie 291171525781777416 [ 478.645282] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 478.663372] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 478.972680] Lustre: server umount lustre-MDT0001 complete [ 483.922772] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 08:31:59 (1788438719) [ 490.678168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.218653] LustreError: 19307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.315293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.554431] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 496.612129] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.733329] LustreError: 19307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.931611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.484867] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 515.696676] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 521.666781] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 527.671727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 533.552591] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 540.953892] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 544.625740] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 551.905137] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.909731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 551.923813] Lustre: Skipped 1 previous similar message [ 551.940522] Lustre: Skipped 6 previous similar messages [ 557.418397] Lustre: server umount lustre-MDT0000 complete [ 561.045960] LustreError: 20041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438797 with bad export cookie 291171525781778298 [ 561.064931] LustreError: 20041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 561.072478] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.454707] Lustre: server umount lustre-MDT0001 complete [ 575.768937] Lustre: server umount lustre-OST0000 complete [ 596.735383] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 605.773528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.182616] LustreError: 23686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.235739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 606.245242] Lustre: Skipped 2 previous similar messages [ 610.770976] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 616.417259] LustreError: 23687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.430544] LustreError: 23687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 618.877312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.121379] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 625.847918] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 632.223261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 638.298912] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 645.186557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 645.414359] Lustre: lustre-OST0001: new disk, initializing [ 645.418322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 645.424116] Lustre: 26000:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 647.073479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 647.082989] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 647.168179] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 651.300467] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 660.185873] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 663.575155] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 675.808489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 675.822967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.843922] Lustre: Skipped 1 previous similar message [ 675.858630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 675.868661] Lustre: Skipped 3 previous similar messages [ 681.518583] Lustre: server umount lustre-MDT0000 complete [ 681.953950] LustreError: 23687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.978987] LustreError: 23687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 685.589086] LustreError: 24427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438922 with bad export cookie 291171525781779145 [ 685.591756] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.601473] LustreError: 24427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 686.093925] Lustre: server umount lustre-MDT0001 complete [ 699.472129] Lustre: server umount lustre-OST0000 complete [ 702.431112] Lustre: 3650:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788438923/real 1788438923] req@ffff89b803978000 x1875313505313280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788438939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 702.458604] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.485535] Lustre: Skipped 3 previous similar messages [ 703.854605] Lustre: server umount lustre-OST0001 complete [ 722.161993] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 733.029456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 733.530501] LustreError: 29025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 739.179911] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 749.359640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.158545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 750.165684] Lustre: Skipped 4 previous similar messages [ 755.641275] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 758.585379] Lustre: 30175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 765.394650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 772.632459] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 782.758887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 792.079295] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 802.270421] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 806.744733] Lustre: 32053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 814.060253] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.063962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 814.083412] Lustre: Skipped 2 previous similar messages [ 814.096333] Lustre: Skipped 7 previous similar messages [ 820.404201] Lustre: server umount lustre-MDT0000 complete [ 824.300983] LustreError: 29026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 824.358683] LustreError: 29026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 825.559758] LustreError: 29006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439062 with bad export cookie 291171525781780440 [ 825.578129] LustreError: 29006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 825.579512] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 826.197790] Lustre: server umount lustre-MDT0001 complete [ 840.526563] Lustre: server umount lustre-OST0000 complete [ 854.983083] Lustre: server umount lustre-OST0001 complete [ 863.139601] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 08:38:18 (1788439098) [ 893.018886] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 904.586500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.059560] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 919.340855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.419609] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 927.384708] Lustre: 36007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 933.406856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 940.202468] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 950.501285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.570672] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 967.351884] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 971.430166] Lustre: 37886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 981.473323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 981.483310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.499612] Lustre: Skipped 1 previous similar message [ 981.506856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 981.514112] Lustre: Skipped 4 previous similar messages [ 983.636461] Lustre: server umount lustre-MDT0000 complete [ 987.117091] LustreError: 34856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.165656] LustreError: 34856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 987.476268] LustreError: 34839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439224 with bad export cookie 291171525781781546 [ 987.483694] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 987.492546] LustreError: 34839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 987.903472] Lustre: server umount lustre-MDT0001 complete [ 1001.753073] Lustre: server umount lustre-OST0000 complete [ 1016.343248] Lustre: server umount lustre-OST0001 complete [ 1025.210934] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 08:41:00 (1788439260) [ 1041.308654] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1051.054718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.516473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.521215] Lustre: Skipped 6 previous similar messages [ 1055.744672] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1064.314813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.631539] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1072.538742] Lustre: 41178:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1079.883584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1087.095863] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1096.094527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1102.991914] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1111.832062] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1116.633438] Lustre: 43059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1117.640974] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1117.650407] Lustre: Skipped 1 previous similar message [ 1123.290903] Lustre: server umount lustre-MDT0000 complete [ 1123.305359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.329529] Lustre: Skipped 5 previous similar messages [ 1127.604340] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439364 with bad export cookie 291171525781782652 [ 1127.618703] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1127.621138] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1134.218557] Lustre: server umount lustre-MDT0001 complete [ 1143.352044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.998818] LustreError: 3647:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff89b8024c0700 x1875313505469696/t0(0) o250->MGC192.168.206.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1157.706627] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1162.670296] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1166.383626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.237177] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1171.973143] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1171.993053] Lustre: Skipped 1 previous similar message [ 1179.655912] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1181.375456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1186.825975] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1188.351346] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1197.540365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1200.021253] Lustre: server umount lustre-MDT0000 complete [ 1203.971147] LustreError: 40012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439440 with bad export cookie 291171525781783744 [ 1203.980654] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1203.985544] LustreError: 40012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1204.525486] Lustre: server umount lustre-MDT0001 complete [ 1219.009251] Lustre: server umount lustre-OST0000 complete [ 1233.367302] Lustre: server umount lustre-OST0001 complete [ 1242.779110] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 08:44:37 (1788439477) [ 1261.521633] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1273.280043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.989141] LustreError: 47666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.013807] LustreError: 47666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 1279.230335] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1287.981641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.127795] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1296.380372] Lustre: 48820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1304.683362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1312.932787] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1322.323606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1329.410681] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1337.603284] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1342.456300] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1355.233117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1355.252715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.263409] Lustre: Skipped 13 previous similar messages [ 1361.151650] Lustre: server umount lustre-MDT0000 complete [ 1365.409532] LustreError: 47646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439602 with bad export cookie 291171525781784570 [ 1365.422365] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1365.426032] LustreError: 47646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1365.713729] Lustre: server umount lustre-MDT0001 complete [ 1381.052902] Lustre: server umount lustre-OST0000 complete [ 1384.866885] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439605/real 1788439605] req@ffff89b803b02a00 x1875313505563520/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1384.917220] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.951586] Lustre: Skipped 10 previous similar messages [ 1385.204468] Lustre: server umount lustre-OST0001 complete [ 1395.517806] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 08:47:10 (1788439630) [ 1397.756909] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1399.554648] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 08:47:14 (1788439634) [ 1410.972674] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 1419.929712] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1445.466171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.908450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1446.047964] Lustre: lustre-MDT0000: new disk, initializing [ 1446.136728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1451.506572] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1460.958600] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1468.979048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.421329] Lustre: lustre-OST0000: new disk, initializing [ 1469.425569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1469.430055] Lustre: Skipped 1 previous similar message [ 1469.434376] Lustre: 54899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1470.964238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1470.983426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1471.010628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1476.129901] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1485.936359] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1488.461181] Lustre: DEBUG MARKER: create test files [ 1522.148060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1524.861688] Lustre: server umount lustre-OST0000 complete [ 1531.199684] Lustre: server umount lustre-MDT0000 complete [ 1545.644285] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 1549.440460] Key type lgssc unregistered [ 1549.818267] LNet: 56529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.829438] LNetError: 56529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.852089] LNet: Removed LNI 192.168.206.143@tcp [ 1551.062325] Key type .llcrypt unregistered [ 1551.065740] Key type ._llcrypt unregistered [ 1581.170398] Key type ._llcrypt registered [ 1581.178302] Key type .llcrypt registered [ 1581.358262] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1582.659426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1582.700387] alg: No test for adler32 (adler32-zlib) [ 1583.922320] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1584.267192] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 1586.065891] Key type lgssc registered [ 1587.888723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1597.827701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1597.856349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1599.287583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1604.930665] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1614.013486] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1621.734779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1622.186044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.238585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1624.321970] Lustre: 59255:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1624.337586] Lustre: 59255:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1624.342478] Lustre: 59255:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1624.352381] Lustre: 59255:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1624.361102] Lustre: 59255:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1624.368130] Lustre: 59255:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1628.115879] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1635.502772] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1640.871491] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1646.622404] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1655.265903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.268478] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1655.277968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.518600] Lustre: server umount lustre-OST0000 complete [ 1660.750728] Lustre: server umount lustre-MDT0000 complete [ 1673.152545] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 1676.105261] Key type lgssc unregistered [ 1676.425680] LNet: 60864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.431497] LNetError: 60864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.442713] LNet: Removed LNI 192.168.206.143@tcp [ 1677.207481] Key type .llcrypt unregistered [ 1677.209565] Key type ._llcrypt unregistered [ 1708.534483] Key type ._llcrypt registered [ 1708.540552] Key type .llcrypt registered [ 1708.725388] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1709.836694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1709.969776] alg: No test for adler32 (adler32-zlib) [ 1711.021609] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1711.295247] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 1712.967339] Key type lgssc registered [ 1714.137290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1722.579260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1722.601901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.245327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.700277] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1736.399968] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1743.848292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1744.225526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.386603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1749.525505] Lustre: 63695:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1749.536052] Lustre: 63695:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1749.540291] Lustre: 63695:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1749.544104] Lustre: 63695:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1749.549518] Lustre: 63695:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1749.554844] Lustre: 63695:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1751.653100] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1760.336630] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1766.035729] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1780.196506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.198808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.216266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.965768] Lustre: server umount lustre-OST0000 complete [ 1786.009796] Lustre: server umount lustre-MDT0000 complete [ 1797.869326] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 1800.725525] Key type lgssc unregistered [ 1800.994705] LNet: 65253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.013948] LNetError: 65253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.038595] LNet: Removed LNI 192.168.206.143@tcp [ 1802.243851] Key type .llcrypt unregistered [ 1802.248519] Key type ._llcrypt unregistered [ 1821.018822] Key type ._llcrypt registered [ 1821.020613] Key type .llcrypt registered [ 1821.146734] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 1837.856821] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 1838.981156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.012989] alg: No test for adler32 (adler32-zlib) [ 1840.318247] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1840.703145] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 1842.463427] Key type lgssc registered [ 1843.681245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1887.240198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1887.315553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.672655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1888.776067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1888.961552] Lustre: lustre-MDT0000: new disk, initializing [ 1889.162907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1889.219807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1895.144298] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1905.863952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.959393] Lustre: 68402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.030728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1906.034893] Lustre: Skipped 1 previous similar message [ 1906.108477] Lustre: lustre-MDT0001: new disk, initializing [ 1906.241694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1906.287014] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1906.312018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1910.942618] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1921.284111] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1927.585470] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1936.523462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1936.910941] Lustre: lustre-OST0000: new disk, initializing [ 1936.915143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1936.928915] Lustre: 70373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1937.057873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.149907] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 1947.210924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1947.217898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1947.311822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1952.403638] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1958.121744] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1958.376297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1963.875987] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1964.059617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1968.098428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1968.105837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1968.112369] LustreError: Skipped 1 previous similar message [ 1968.135314] Lustre: Skipped 1 previous similar message [ 1968.138811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1968.149580] Lustre: Skipped 1 previous similar message [ 1973.219520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.238080] Lustre: Skipped 1 previous similar message [ 1978.359502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.373185] Lustre: Skipped 1 previous similar message [ 1980.385208] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1980.641492] Lustre: server umount lustre-OST0000 complete [ 1988.064312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1988.071215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.085051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.698384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.703911] Lustre: Skipped 2 previous similar messages [ 1999.327213] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1999.478877] Lustre: server umount lustre-MDT0000 complete [ 2003.162902] LustreError: 68394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440240 with bad export cookie 485423003559468187 [ 2003.168192] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2003.170448] LustreError: 68394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2003.442763] Lustre: server umount lustre-MDT0001 complete [ 2011.752937] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 08:57:26 (1788440246) [ 2022.884068] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2062.766832] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 2074.326755] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2110.634119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.946140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2110.980770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2111.097347] Lustre: lustre-MDT0000: new disk, initializing [ 2111.186574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.203739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2115.251538] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2126.141782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.280912] Lustre: 76393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2126.323744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2126.327338] Lustre: Skipped 1 previous similar message [ 2126.388166] Lustre: lustre-MDT0001: new disk, initializing [ 2126.468460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2126.506993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2126.523120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2131.100297] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2140.666489] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2146.817629] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2153.379446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2153.624162] Lustre: lustre-OST0000: new disk, initializing [ 2153.629809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2153.634908] Lustre: 78361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2153.765775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.183410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2159.207221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2159.364410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2159.818206] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2170.668810] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2176.724697] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2176.960720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2181.997876] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2182.176732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2185.184394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2185.196754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.214170] LustreError: Skipped 1 previous similar message [ 2185.215518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2185.220579] Lustre: Skipped 2 previous similar messages [ 2185.237818] Lustre: Skipped 3 previous similar messages [ 2190.852847] Lustre: server umount lustre-OST0000 complete [ 2195.424607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.494047] LustreError: 79905:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000fbe5b2bb) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 2203.600327] Lustre: server umount lustre-MDT0000 complete [ 2207.696203] LustreError: 76387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440444 with bad export cookie 485423003559469069 [ 2207.710634] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2207.715252] LustreError: 76387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2208.306265] Lustre: server umount lustre-MDT0001 complete [ 2218.640577] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 09:00:53 (1788440453) [ 2227.151736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.848509] LustreError: 80791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.876635] LustreError: 80791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2227.946480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.312843] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.435933] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2237.410550] LustreError: 80791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.515110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.529481] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.022504] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2256.059202] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2261.533565] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2268.701075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2269.118921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.124264] Lustre: Skipped 1 previous similar message [ 2275.886045] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2282.929363] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2286.313516] Lustre: 83464:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting b6928929-0e99-41e8-91c7-c6191f6bc243 at adminstrative request [ 2287.522589] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.540084] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2297.834985] Lustre: MGS: Client b6928929-0e99-41e8-91c7-c6191f6bc243 (at 0@lo) reconnecting [ 2297.845246] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x6bc91a65cf7237f to 0x6bc91a65cf7261f [ 2297.869388] Lustre: MGC192.168.206.143@tcp: Connection restored to 0@lo (at 0@lo) [ 2324.972548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.252303] Lustre: lustre-OST0001: new disk, initializing [ 2325.258832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2325.264940] Lustre: 83861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2331.757716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2331.795744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2331.949703] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2331.963106] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2342.573183] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2347.503978] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2347.504346] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.512969] LustreError: Skipped 1 previous similar message [ 2347.522331] Lustre: Skipped 2 previous similar messages [ 2347.535041] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2347.541754] Lustre: Skipped 6 previous similar messages [ 2351.534212] Lustre: server umount lustre-OST0001 complete [ 2352.614693] LustreError: 82613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.641521] LustreError: 82613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2357.734676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.002911] Lustre: server umount lustre-OST0000 complete [ 2372.855485] Lustre: server umount lustre-MDT0000 complete [ 2373.088760] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.121065] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2377.613869] LustreError: 80771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440614 with bad export cookie 485423003559470623 [ 2377.623923] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2377.638993] LustreError: 80771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2378.182589] Lustre: server umount lustre-MDT0001 complete [ 2387.691601] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 09:03:42 (1788440622) [ 2389.539919] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2391.579750] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 09:03:46 (1788440626) [ 2393.212723] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2395.093605] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 09:03:50 (1788440630) [ 2397.131951] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2399.266477] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 09:03:54 (1788440634) [ 2400.971995] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2402.840328] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 09:03:57 (1788440637) [ 2404.560293] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2410.822439] loop: module loaded [ 2410.841607] loop0: detected capacity change from 0 to 4294967296 [ 2410.899457] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2412.358418] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2415.937362] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2419.810408] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2446.859420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.903200] LustreError: 87984:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2446.912352] LustreError: 87984:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2446.918406] LustreError: 87984:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2446.925572] LustreError: 87984:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2468.432806] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 2479.236601] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 2522.482102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.742153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2522.803425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2522.993722] Lustre: lustre-MDT0000: new disk, initializing [ 2523.088842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2523.095329] Lustre: Skipped 1 previous similar message [ 2523.115310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2529.103706] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2541.084970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.285067] Lustre: 90857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2541.586566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2546.926387] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2556.881645] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2563.129661] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2570.087247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.348470] Lustre: 92825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2576.503436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2576.661048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2578.748830] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2589.074925] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2595.121406] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2595.395554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2600.978626] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2601.300934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2607.591448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2607.592442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.607262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2607.624483] Lustre: Skipped 4 previous similar messages [ 2607.649299] Lustre: Skipped 6 previous similar messages [ 2618.335419] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2618.635838] Lustre: server umount lustre-OST0000 complete [ 2637.279206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2637.494285] Lustre: server umount lustre-MDT0000 complete [ 2638.305116] LustreError: 90864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.319958] LustreError: 90864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2641.467421] LustreError: 90847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440878 with bad export cookie 485423003559471239 [ 2641.471661] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2641.478680] LustreError: 90847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2642.016613] Lustre: server umount lustre-MDT0001 complete [ 2651.815356] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 09:08:06 (1788440886) [ 2659.095959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.662399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2659.668831] Lustre: Skipped 2 previous similar messages [ 2665.535807] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2675.168745] LustreError: 95242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.195262] LustreError: 95242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2675.925939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.510446] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2689.942504] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2696.197949] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2703.233613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2710.311410] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2718.662918] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2723.824580] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2724.077862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2729.304572] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2729.531524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2736.963881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2737.141436] Lustre: lustre-OST0001: new disk, initializing [ 2737.146397] Lustre: Skipped 2 previous similar messages [ 2737.151690] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2737.156779] Lustre: Skipped 3 previous similar messages [ 2737.163397] Lustre: 98612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2744.435203] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2747.466639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2747.474143] Lustre: Skipped 2 previous similar messages [ 2747.484016] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2747.572852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2755.409274] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2760.818947] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2761.142813] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2766.432656] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2766.757768] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2792.806761] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2820.709128] Lustre: Failing over lustre-MDT0000 [ 2821.214273] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.43@tcp (stopping) [ 2821.219558] Lustre: Skipped 10 previous similar messages [ 2821.400891] Lustre: server umount lustre-MDT0000 complete [ 2824.679011] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2824.683855] LustreError: 95238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.710658] Lustre: Skipped 4 previous similar messages [ 2824.753443] LustreError: 95238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2841.061810] Lustre: 66627:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441061/real 1788441061] req@0000000019076046 x1875315342467712/t0(0) o400->MGC192.168.206.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441077 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2841.088295] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2849.292933] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2849.296322] LDISKFS-fs (dm-0): recovery complete [ 2849.301976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.297702] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6bc91a65cf7f093 [ 2850.306444] Lustre: MGC192.168.206.143@tcp: Connection restored to 0@lo (at 0@lo) [ 2850.643591] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2852.925228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2852.927853] Lustre: 99285:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2854.581804] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 2855.914327] Lustre: 95243:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2855.928927] Lustre: 95243:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2855.944182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2856.056276] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2856.396049] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2856.740862] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2856.757485] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2857.087122] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2857.103780] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2857.118839] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2857.139956] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2857.785154] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2857.824087] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2858.143661] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2858.165188] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2859.160416] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2859.179610] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 5 previous similar messages [ 2859.825085] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2859.846331] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2860.175956] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2860.192680] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 5 previous similar messages [ 2863.396270] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2863.413563] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2864.055500] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2864.078939] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2864.399104] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2864.414163] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2871.705872] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2871.723111] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 25 previous similar messages [ 2872.361614] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2872.373529] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 25 previous similar messages [ 2872.695163] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2872.709851] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 25 previous similar messages [ 2887.781674] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2887.815179] Lustre: 101465:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 49 previous similar messages [ 2888.467535] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2888.487358] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 2888.815178] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2888.822976] LustreError: 101465:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 49 previous similar messages [ 2912.264841] Lustre: 101465:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2912.283567] Lustre: 101465:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2912.294567] Lustre: 101465:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 15e962d2-ab82-4de2-a448-a9ea4798a96c@192.168.206.43@tcp [ 2912.310660] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2912.324621] Lustre: 101465:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@000000009413e409 x1875315325473664/t0(8589934772) o35->15e962d2-ab82-4de2-a448-a9ea4798a96c@192.168.206.43@tcp:179/0 lens 392/0 e 0 to 0 dl 1788441159 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2912.393927] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2912.394055] Lustre: lustre-MDT0000: Denying connection for new client 15e962d2-ab82-4de2-a448-a9ea4798a96c (at 192.168.206.43@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2912.447575] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2912.506831] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2912.514481] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2912.538471] Lustre: Skipped 2 previous similar messages [ 2912.576923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 2912.577310] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:46 to 0x2c0000401:545) [ 2927.593654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.611037] Lustre: Skipped 2 previous similar messages [ 2932.909252] Lustre: server umount lustre-OST0000 complete [ 2937.317871] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2937.325184] LustreError: Skipped 1 previous similar message [ 2943.617968] Lustre: server umount lustre-OST0001 complete [ 2953.185476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2953.196333] Lustre: Skipped 12 previous similar messages [ 2962.400570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2962.747974] Lustre: server umount lustre-MDT0000 complete [ 2963.427154] LustreError: 97617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.459674] LustreError: 97617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 41 previous similar messages [ 2967.536792] LustreError: 95223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441204 with bad export cookie 485423003559522451 [ 2967.566164] LustreError: 95223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2967.570102] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2968.101114] Lustre: server umount lustre-MDT0001 complete [ 2980.565257] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 09:13:35 (1788441215) [ 2989.576310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2989.586653] Lustre: *** cfs_fail_loc=197, val=0*** [ 3005.217485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3010.335573] LustreError: 103450:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.143@tcp: failed processing log, type 4: rc = -110 [ 3015.499378] Lustre: *** cfs_fail_loc=197, val=0*** [ 3015.505019] Lustre: Skipped 65 previous similar messages [ 3036.063346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.073751] Lustre: Skipped 4 previous similar messages [ 3041.937592] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3048.932710] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3051.431639] Lustre: server umount lustre-OST0000 complete [ 3059.988810] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 09:14:55 (1788441295) [ 3081.711543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.843216] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3096.137465] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3100.458673] LustreError: 105100:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3100.477577] LustreError: 105100:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3100.488552] LustreError: 105100:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 3100.925391] Lustre: server umount lustre-MDT0000 complete [ 3132.815447] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 3143.727391] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3183.064869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.443499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3183.485790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3183.536762] Lustre: lustre-MDT0000: new disk, initializing [ 3183.603744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3188.038917] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3198.861969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.999965] Lustre: 109278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3199.337628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3203.674745] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3213.563885] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3219.332526] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3225.292591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3225.521096] Lustre: 111247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3231.528701] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3233.946289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3241.033563] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3245.587810] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3245.832157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3250.564440] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3250.900899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3254.242080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3254.253070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.265274] Lustre: Skipped 4 previous similar messages [ 3254.271073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3254.277534] Lustre: Skipped 2 previous similar messages [ 3258.922966] Lustre: server umount lustre-OST0000 complete [ 3269.839361] Lustre: server umount lustre-MDT0000 complete [ 3270.112437] LustreError: 109289:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.138041] LustreError: 109289:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3274.563037] LustreError: 109269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441511 with bad export cookie 485423003559533070 [ 3274.578727] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3275.003496] Lustre: server umount lustre-MDT0001 complete [ 3284.026424] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 09:18:39 (1788441519) [ 3292.130343] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 3294.790847] Key type lgssc unregistered [ 3295.138553] LNet: 113734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3295.148037] LNetError: 113734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3296.166456] LNet: Removed LNI 192.168.206.143@tcp [ 3297.021907] Key type .llcrypt unregistered [ 3297.026581] Key type ._llcrypt unregistered [ 3314.493389] Key type ._llcrypt registered [ 3314.495525] Key type .llcrypt registered [ 3314.649095] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 3330.072855] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3331.001166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3331.105612] alg: No test for adler32 (adler32-zlib) [ 3332.159473] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3332.421367] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 3334.079202] Key type lgssc registered [ 3335.144980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3376.382502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3376.433703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.714476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3377.755988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3377.842792] Lustre: lustre-MDT0000: new disk, initializing [ 3377.991449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.023905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3382.186277] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3393.699611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3393.976983] Lustre: lustre-OST0000: new disk, initializing [ 3393.981459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3393.988308] Lustre: Skipped 1 previous similar message [ 3393.993340] Lustre: 117902:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3394.072212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3394.798074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3394.804605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3394.840837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3401.097788] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3412.778691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3413.012917] Lustre: lustre-OST0001: new disk, initializing [ 3413.016379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3413.022117] Lustre: 118766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3413.133345] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3419.448741] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3421.816434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3421.824390] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3421.848361] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3431.922238] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.932189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3431.940275] Lustre: Skipped 1 previous similar message [ 3431.954738] Lustre: Skipped 1 previous similar message [ 3434.429952] Lustre: server umount lustre-MDT0000 complete [ 3440.987229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3447.338761] LustreError: 117090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441684 with bad export cookie 5045256071035453769 [ 3447.342732] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3447.361653] LustreError: 117090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3447.560961] Lustre: server umount lustre-OST0000 complete [ 3452.422667] Lustre: server umount lustre-OST0001 complete [ 3465.109113] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 09:21:39 (1788441699) [ 3488.323513] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 09:22:03 (1788441723) [ 3498.626315] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3522.349880] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 09:22:37 (1788441757) [ 3560.862166] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 3569.681441] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3600.617929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3600.816158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3600.835757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3600.890424] Lustre: lustre-MDT0000: new disk, initializing [ 3600.938029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.959806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3605.008475] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3615.121933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3615.242494] Lustre: 126522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3615.322682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3615.328210] Lustre: Skipped 1 previous similar message [ 3615.416744] Lustre: lustre-MDT0001: new disk, initializing [ 3615.513959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3615.554073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3615.571969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3620.314298] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3630.187673] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3636.523846] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3643.392370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3643.730605] Lustre: lustre-OST0000: new disk, initializing [ 3643.733916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3643.741700] Lustre: 128493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3643.834071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3650.598045] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3653.195044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3653.207661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3653.356360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3660.772860] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3665.120218] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3665.389239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3669.909423] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3670.142430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3674.086621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3674.087963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.108863] LustreError: Skipped 1 previous similar message [ 3674.134169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3674.140712] Lustre: Skipped 1 previous similar message [ 3679.201551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3679.210049] Lustre: Skipped 1 previous similar message [ 3684.327514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3684.337757] Lustre: Skipped 1 previous similar message [ 3686.367365] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3686.568854] Lustre: server umount lustre-OST0000 complete [ 3692.514471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3692.524214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.534109] Lustre: Skipped 1 previous similar message [ 3692.543925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3694.574901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.320412] Lustre: server umount lustre-MDT0000 complete [ 3700.055360] LustreError: 126513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441936 with bad export cookie 5045256071035454994 [ 3700.072288] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3700.432497] Lustre: server umount lustre-MDT0001 complete [ 3709.466795] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 09:25:44 (1788441944) [ 3716.384810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.754833] LustreError: 130904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.785981] LustreError: 130904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3716.857399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3721.720450] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3722.213893] LustreError: 130905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3726.310810] LustreError: 130904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3730.322944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3734.956614] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3743.269948] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3749.079473] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3755.354134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3755.433166] LustreError: 132729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3755.714545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3755.721988] Lustre: Skipped 1 previous similar message [ 3762.382843] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3769.516935] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3773.750875] Lustre: *** cfs_fail_loc=159, val=0*** [ 3789.431235] Lustre: lustre-MDT0000: Client c7f8e6ca-ceb4-4c50-a7e7-304e2c11c4f0 (at 192.168.206.43@tcp) reconnecting [ 3790.920242] Lustre: *** cfs_fail_loc=159, val=0*** [ 3790.928434] Lustre: Skipped 5 previous similar messages [ 3807.356860] Lustre: lustre-MDT0000: Client c7f8e6ca-ceb4-4c50-a7e7-304e2c11c4f0 (at 192.168.206.43@tcp) reconnecting [ 3827.687415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3827.697198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3827.717773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.725855] Lustre: Skipped 2 previous similar messages [ 3830.377839] Lustre: server umount lustre-OST0000 complete [ 3838.432099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3838.439038] LustreError: Skipped 1 previous similar message [ 3838.447163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.461548] Lustre: Skipped 1 previous similar message [ 3840.212527] Lustre: server umount lustre-MDT0000 complete [ 3843.771134] LustreError: 130884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442080 with bad export cookie 5045256071035455876 [ 3843.774283] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3843.787965] LustreError: 130884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3844.327516] Lustre: server umount lustre-MDT0001 complete [ 3855.627443] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 3858.415332] Key type lgssc unregistered [ 3858.705728] LNet: 134682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3858.710299] LNetError: 134682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3858.725646] LNet: Removed LNI 192.168.206.143@tcp [ 3859.698336] Key type .llcrypt unregistered [ 3859.700782] Key type ._llcrypt unregistered [ 3884.020228] Key type ._llcrypt registered [ 3884.025886] Key type .llcrypt registered [ 3884.137046] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 3885.239221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3885.453510] alg: No test for adler32 (adler32-zlib) [ 3886.459662] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3886.664169] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 3888.391681] Key type lgssc registered [ 3889.738642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3897.098654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3897.112976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.617962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.053821] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3909.098458] LustreError: 136096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3909.113184] LustreError: 136096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3911.680781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.121567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3917.675551] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3927.505190] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3933.807214] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3940.292572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3940.624586] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3943.739394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3946.835594] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 3953.803522] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3966.200138] Lustre: *** cfs_fail_loc=159, val=0*** [ 3981.415790] Lustre: lustre-MDT0000: Client 65fbd78e-bf62-4528-84f8-199df3cd0f1c (at 192.168.206.43@tcp) reconnecting [ 3983.313680] Lustre: *** cfs_fail_loc=159, val=0*** [ 3999.849630] Lustre: lustre-MDT0001: Client 65fbd78e-bf62-4528-84f8-199df3cd0f1c (at 192.168.206.43@tcp) reconnecting [ 4001.369210] Lustre: *** cfs_fail_loc=159, val=0*** [ 4001.377191] Lustre: Skipped 3 previous similar messages [ 4017.876714] Lustre: lustre-MDT0001: Client 65fbd78e-bf62-4528-84f8-199df3cd0f1c (at 192.168.206.43@tcp) reconnecting [ 4022.737410] Lustre: 139141:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4025.662584] Lustre: *** cfs_fail_loc=159, val=0*** [ 4025.664341] Lustre: Skipped 4 previous similar messages [ 4041.831509] Lustre: lustre-MDT0000: Client c1ab2b96-4c26-4838-9647-87bffa18c301 (at 192.168.206.43@tcp) reconnecting [ 4043.393502] Lustre: *** cfs_fail_loc=159, val=0*** [ 4043.404188] Lustre: Skipped 14 previous similar messages [ 4059.765853] Lustre: lustre-MDT0000: Client c1ab2b96-4c26-4838-9647-87bffa18c301 (at 192.168.206.43@tcp) reconnecting [ 4061.175254] Lustre: 139433:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4080.096672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4080.105808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.123355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.144035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4082.149882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.161167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.171501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.181480] Lustre: Skipped 1 previous similar message [ 4085.508618] Lustre: server umount lustre-OST0000 complete [ 4091.360463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4091.367306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.383344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.418405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.787972] Lustre: server umount lustre-MDT0000 complete [ 4098.533019] LustreError: 139202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.547835] LustreError: 139202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4099.803754] LustreError: 139263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442336 with bad export cookie 277688783890655491 [ 4099.805930] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4099.823959] LustreError: 139263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4100.192326] Lustre: server umount lustre-MDT0001 complete [ 4111.559683] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 4115.362916] Key type lgssc unregistered [ 4115.783461] LNet: 140513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4115.802573] LNetError: 140513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4115.849963] LNet: Removed LNI 192.168.206.143@tcp [ 4117.036845] Key type .llcrypt unregistered [ 4117.038448] Key type ._llcrypt unregistered [ 4141.208519] Key type ._llcrypt registered [ 4141.214415] Key type .llcrypt registered [ 4141.347614] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4142.245539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4142.306201] alg: No test for adler32 (adler32-zlib) [ 4143.354398] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4143.572153] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 4145.223147] Key type lgssc registered [ 4146.152360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4153.766516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4153.779849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.213285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.977709] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4165.613439] LustreError: 141924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4165.636063] LustreError: 141924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4167.983427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.386077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4173.060226] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4180.084830] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4186.070729] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4192.326698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4192.616857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4193.648444] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4199.140603] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4200.838865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4207.390179] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4213.626111] Lustre: 144509:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4217.173399] Lustre: 144557:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4219.006263] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4220.085725] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4224.081774] Lustre: 144705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4225.197829] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4226.252440] Lustre: 144813:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4231.656360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4231.659051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.670056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4231.699023] Lustre: Skipped 1 previous similar message [ 4236.573040] Lustre: server umount lustre-OST0000 complete [ 4241.892724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.897207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.913629] Lustre: Skipped 1 previous similar message [ 4241.925429] Lustre: Skipped 1 previous similar message [ 4246.993732] Lustre: server umount lustre-MDT0000 complete [ 4247.014183] LustreError: 141923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4247.049673] LustreError: 141923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4251.312686] LustreError: 141905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442488 with bad export cookie 1379544757025241924 [ 4251.327970] LustreError: 141905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4251.332985] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4251.738504] Lustre: server umount lustre-MDT0001 complete [ 4262.665279] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 4265.703153] Key type lgssc unregistered [ 4265.964948] LNet: 145869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4265.969429] LNetError: 145869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4265.985122] LNet: Removed LNI 192.168.206.143@tcp [ 4266.880584] Key type .llcrypt unregistered [ 4266.884118] Key type ._llcrypt unregistered [ 4289.079255] Key type ._llcrypt registered [ 4289.082444] Key type .llcrypt registered [ 4289.179356] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4290.404967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4290.454102] alg: No test for adler32 (adler32-zlib) [ 4291.539740] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4291.792729] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 4293.455184] Key type lgssc registered [ 4294.897685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4303.007134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4303.029736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4304.497674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4308.768107] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4315.111269] LustreError: 147280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4315.135720] LustreError: 147280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4317.143750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.569068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4322.301387] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4330.099508] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4335.949049] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4343.187195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4343.498182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.317642] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4349.678557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4356.104308] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4362.994661] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4365.636538] Lustre: *** cfs_fail_loc=159, val=0*** [ 4382.328414] Lustre: lustre-MDT0000: Client 506866f0-c7a1-45c6-99a3-c5a3000ca8e3 (at 192.168.206.43@tcp) reconnecting [ 4388.833642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4388.840922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.848545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4388.892016] Lustre: Skipped 1 previous similar message [ 4393.958807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4393.965716] Lustre: Skipped 2 previous similar messages [ 4394.388915] Lustre: server umount lustre-OST0000 complete [ 4399.075949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.095618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.109093] Lustre: Skipped 1 previous similar message [ 4404.192416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4404.208212] Lustre: Skipped 1 previous similar message [ 4404.835734] Lustre: server umount lustre-MDT0000 complete [ 4409.198392] LustreError: 147261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442646 with bad export cookie 6501916399438417933 [ 4409.208571] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4409.217258] LustreError: 147261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4409.571554] Lustre: server umount lustre-MDT0001 complete [ 4421.637376] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 4425.094621] Key type lgssc unregistered [ 4425.406682] LNet: 150940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4425.415565] LNetError: 150940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4426.472812] LNet: Removed LNI 192.168.206.143@tcp [ 4427.673032] Key type .llcrypt unregistered [ 4427.676391] Key type ._llcrypt unregistered [ 4452.978870] Key type ._llcrypt registered [ 4452.986485] Key type .llcrypt registered [ 4453.179700] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4454.447270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4454.523795] alg: No test for adler32 (adler32-zlib) [ 4455.633150] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4455.865065] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 4457.503220] Key type lgssc registered [ 4458.814465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4468.789951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4468.828818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4470.374986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4474.980212] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4480.482854] LustreError: 152350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4480.499522] LustreError: 152350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4484.407491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.969210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4490.250914] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4498.222830] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4504.347392] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4511.941275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4512.277225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4515.383026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4519.326813] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4527.035645] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4529.735909] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4552.178262] Lustre: lustre-OST0000: evicting 0daee344-2d33-4f17-a7c7-bd6cd8ba3bf7 (at 192.168.206.43@tcp) by administrative request [ 4552.185327] Lustre: Skipped 1 previous similar message [ 4580.840958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.847387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4580.848945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4580.862862] Lustre: Skipped 1 previous similar message [ 4585.965490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4585.973749] Lustre: Skipped 2 previous similar messages [ 4586.834901] Lustre: server umount lustre-OST0000 complete [ 4591.076882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.084368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4591.097553] Lustre: Skipped 1 previous similar message [ 4591.116221] Lustre: Skipped 1 previous similar message [ 4596.194467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4596.204713] Lustre: Skipped 1 previous similar message [ 4597.297234] Lustre: server umount lustre-MDT0000 complete [ 4601.308618] LustreError: 152331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442838 with bad export cookie 1529146282613610776 [ 4601.315261] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4601.326586] LustreError: 152331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4601.761485] Lustre: server umount lustre-MDT0001 complete [ 4612.829584] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 4615.532538] Key type lgssc unregistered [ 4615.800578] LNet: 156421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4615.815653] LNetError: 156421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4615.830942] LNet: Removed LNI 192.168.206.143@tcp [ 4616.740442] Key type .llcrypt unregistered [ 4616.742324] Key type ._llcrypt unregistered [ 4695.425337] Key type ._llcrypt registered [ 4695.430137] Key type .llcrypt registered [ 4695.536282] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4696.540797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4696.681624] alg: No test for adler32 (adler32-zlib) [ 4697.919982] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4698.145846] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 4699.807210] Key type lgssc registered [ 4700.966305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4709.302505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4709.326903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4710.834993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.021787] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4721.127454] LustreError: 158911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4721.165260] LustreError: 158911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4724.052964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4724.637257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4729.206586] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4736.957946] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4742.807136] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4750.629154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4750.957481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4757.119102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4757.594975] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4764.979897] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4769.396423] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4777.954651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4777.965669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.982113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4780.000119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4780.009749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4780.015791] LustreError: Skipped 1 previous similar message [ 4780.015815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4780.059940] Lustre: Skipped 1 previous similar message [ 4782.734557] Lustre: server umount lustre-OST0000 complete [ 4790.243342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4790.246966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4790.255030] Lustre: Skipped 1 previous similar message [ 4790.266546] Lustre: Skipped 1 previous similar message [ 4792.660500] Lustre: server umount lustre-MDT0000 complete [ 4795.364566] LustreError: 158906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4795.392342] LustreError: 158906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4796.545178] LustreError: 158893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443033 with bad export cookie 5443053622402787910 [ 4796.549842] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4796.554636] LustreError: 158893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4797.012203] Lustre: server umount lustre-MDT0001 complete [ 4808.967835] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 4811.848156] Key type lgssc unregistered [ 4812.178982] LNet: 162451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4812.192568] LNetError: 162451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4812.216479] LNet: Removed LNI 192.168.206.143@tcp [ 4813.106139] Key type .llcrypt unregistered [ 4813.109100] Key type ._llcrypt unregistered [ 4838.941402] Key type ._llcrypt registered [ 4838.944547] Key type .llcrypt registered [ 4839.091279] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4840.158556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4840.214030] alg: No test for adler32 (adler32-zlib) [ 4841.419597] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4841.720413] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 4843.447207] Key type lgssc registered [ 4844.909644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4862.628648] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 09:44:57 (1788443097) [ 4881.805385] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 4893.742287] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 4931.461092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4931.480842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4932.757629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4932.778742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4932.873329] Lustre: lustre-MDT0000: new disk, initializing [ 4933.026917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4933.043970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4937.729817] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4947.966963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.073900] Lustre: 166718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4948.134657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4948.138996] Lustre: Skipped 1 previous similar message [ 4948.205241] Lustre: lustre-MDT0001: new disk, initializing [ 4948.284112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4948.332081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4948.339306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4952.596343] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4961.802592] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4967.908182] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4974.222860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4974.480425] Lustre: lustre-OST0000: new disk, initializing [ 4974.484566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4974.491173] Lustre: 168686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4974.574443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4980.865466] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 4982.851912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4982.863671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4982.920329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4989.833146] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5003.745111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5003.754191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5003.765418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5003.778404] Lustre: Skipped 1 previous similar message [ 5005.387487] Lustre: server umount lustre-OST0000 complete [ 5008.865749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.870533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5008.877481] Lustre: Skipped 1 previous similar message [ 5008.882568] Lustre: Skipped 1 previous similar message [ 5013.990258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5014.002406] Lustre: Skipped 2 previous similar messages [ 5019.104690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5019.113475] Lustre: Skipped 1 previous similar message [ 5023.202032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5023.205860] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 5023.210919] Lustre: Skipped 1 previous similar message [ 5023.440148] Lustre: server umount lustre-MDT0000 complete [ 5026.748829] LustreError: 168685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443263 with bad export cookie 342395049663575276 [ 5026.764195] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5026.773323] LustreError: 168685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5027.074224] Lustre: server umount lustre-MDT0001 complete [ 5037.624277] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 5040.353274] Key type lgssc unregistered [ 5040.715767] LNet: 170588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5040.736382] LNetError: 170588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5040.757190] LNet: Removed LNI 192.168.206.143@tcp [ 5041.568168] Key type .llcrypt unregistered [ 5041.570809] Key type ._llcrypt unregistered [ 5065.405870] Key type ._llcrypt registered [ 5065.410101] Key type .llcrypt registered [ 5065.497753] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 5066.439041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5066.461995] alg: No test for adler32 (adler32-zlib) [ 5067.612921] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5067.853436] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 5069.559158] Key type lgssc registered [ 5070.599804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5077.610791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5077.640883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5079.104084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5083.713675] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5088.232070] LustreError: 172001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.246296] LustreError: 172001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5092.793423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5093.218843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5097.993946] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5105.769117] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5112.254774] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5119.973336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5120.377240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.460893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5127.417560] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 5135.340297] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5144.422287] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5144.451364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.463040] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5144.472412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.487819] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5145.558206] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5145.582890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.600498] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5145.611513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.630361] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5146.809704] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5146.824239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.835069] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5146.838981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.846075] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5147.813403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.746628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.575904] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5149.582519] Lustre: Skipped 2 previous similar messages [ 5149.599296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.604917] Lustre: Skipped 2 previous similar messages [ 5149.610904] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5149.620123] Lustre: Skipped 2 previous similar messages [ 5149.628277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.636440] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5149.643854] Lustre: Skipped 2 previous similar messages [ 5150.623246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.455034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.498312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.453827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.681810] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5154.691513] Lustre: Skipped 4 previous similar messages [ 5154.722881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5154.733473] Lustre: Skipped 4 previous similar messages [ 5154.742783] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5154.752465] Lustre: Skipped 4 previous similar messages [ 5154.762179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.776143] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5154.788870] Lustre: Skipped 4 previous similar messages [ 5155.726225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.647816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.486681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.348648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5159.241140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.220102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.168219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.057937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.979843] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5162.990501] Lustre: Skipped 8 previous similar messages [ 5163.013207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5163.020932] Lustre: Skipped 8 previous similar messages [ 5163.027132] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5163.036187] Lustre: Skipped 8 previous similar messages [ 5163.041819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.060231] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5163.074676] Lustre: Skipped 8 previous similar messages [ 5163.965621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.043362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.033674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.041037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.913336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.707306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.505031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.508033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.311143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.088211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.008162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5173.823738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.715140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.532682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.250772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.018720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.778764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5178.526818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.346470] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5179.354624] Lustre: Skipped 18 previous similar messages [ 5179.374369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5179.385932] Lustre: Skipped 18 previous similar messages [ 5179.392113] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5179.400662] Lustre: Skipped 18 previous similar messages [ 5179.404756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.415321] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5179.427012] Lustre: Skipped 18 previous similar messages [ 5180.261789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.210116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.046996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5182.765500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.560487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.236094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.975079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.893559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.852887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5187.783098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.695451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.570623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.442115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.374739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.217640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.084469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.916941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.858576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.908644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.655704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.679443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.545268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.515329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.526116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.378034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.152413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.995566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.903572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.859281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.718513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.784969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.689428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.587647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.599361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.618520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.596466] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5211.601514] Lustre: Skipped 35 previous similar messages [ 5211.613542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.623768] Lustre: Skipped 35 previous similar messages [ 5211.628679] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5211.643288] Lustre: Skipped 35 previous similar messages [ 5211.647587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.656434] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5211.663336] Lustre: Skipped 35 previous similar messages [ 5212.612738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.695473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.752959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.738972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.590365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.470643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.484689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.308207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.995461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.747177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.468954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.251478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.100619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.005242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.934199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.865990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.739677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.605959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.514788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.552628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.483462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.451825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.400907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.337388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.375926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.437945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.518458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.395513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.282608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5239.140217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.132698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.998591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.885815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.694551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.570179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.590251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.548407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.341966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5247.269843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.004722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.786562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.649608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.558706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.412465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.089822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.958820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.785846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.609809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.311443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.046917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.905297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.749645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.501679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.350576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.336435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.356776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.225982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.114067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.891237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.644742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.382880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.180395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.944628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.837151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.679877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.680365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.467095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.448213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.291056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.345304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.255822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.122547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.841594] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5275.848130] Lustre: Skipped 72 previous similar messages [ 5275.862626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.873904] Lustre: Skipped 72 previous similar messages [ 5275.892127] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5275.898798] Lustre: Skipped 72 previous similar messages [ 5275.906538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.917041] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5275.932189] Lustre: Skipped 72 previous similar messages [ 5276.818894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.750116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.634482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.481076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.464901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.420940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.351389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.179926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.999065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.800705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.638419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.414681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.360420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.377920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.332623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.121249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.014157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.831382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.900643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.991774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.026450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.054724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.259935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.315106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.420572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.328320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.209376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.188301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.226384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.113601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.017851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.036974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.045916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.947956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.839991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.739927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.637051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.632420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.617686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.706987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.661716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.449827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.497835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.389523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.275714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.199780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.391748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.646369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.618685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.817589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.864155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.093665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.363742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.367831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.474481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.392989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.316267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.268028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.343237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.350756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.370829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.395268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.687565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.824371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.027610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.363619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.268346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.181258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.216231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.078099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.020831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.973424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.991719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.007287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.928349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5350.909393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.040768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.171680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.246039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.170410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.279626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5357.341198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.351951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.373596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.328115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.461026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.353188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.411597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.375799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.336668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5366.228258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.118323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.949478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.652636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.652108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.668745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.476214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.531836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.507694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.453288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.267358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.455714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.268763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.138491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.044306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.146604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.232167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.371696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.396984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.415741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.479751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.711956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.646574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.500732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.409231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.489565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.765855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.715594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.816391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.942152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.970487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.146704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.383234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.350821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.358424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.233732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.178543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.087303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.930903] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5403.941570] Lustre: Skipped 128 previous similar messages [ 5403.963478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.979462] Lustre: Skipped 128 previous similar messages [ 5403.984307] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5403.989348] Lustre: Skipped 128 previous similar messages [ 5403.992901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.002114] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5404.012110] Lustre: Skipped 128 previous similar messages [ 5404.775043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.607979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.475280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.464580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.283909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.552819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.703281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.573975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.464122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.402692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.304698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.312156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.186303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.112389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.992902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.886549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.788950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.774106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.643573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.449819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.402467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.186394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.032985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.997810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.976057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.980148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.940759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.989378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.260172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.230584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.123568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.073983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.986551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.245566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.425587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.511244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.932243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.531151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.709712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.163415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.289925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.410022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.686240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.077581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.365665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.647368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.711048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.698466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.798584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.970967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.958539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.188084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.145362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.213584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.232073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.294580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.537845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.665755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.625437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.540628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.466822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.460439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.326454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.394824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.311225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.204654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.279277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.285127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.144851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.974697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.976652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.869072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.942212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.894681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.879691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.817345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.711883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.779364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.827332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.837643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.021258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.982246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.798646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.673977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.595698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.486569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.553410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.520424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.339675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.113963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.016918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.725701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.538953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.477832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.321481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.107305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.977677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.820805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.607977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.438063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.229854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.990877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.760959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.523395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.306581] LustreError: 172021:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5507.313991] LustreError: 172021:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5507.327159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.076733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.917726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.185806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.923093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.766239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.378926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.136517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.831107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.763514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.721809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.518264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.270414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.285782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.996664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.004651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.807762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.740558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.499943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.311753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.261208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.214989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.145247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.252389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.277252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.169483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.949947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.074472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.006596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.889383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.735139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.656110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.544323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5536.508125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.409773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.238922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.184675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.117032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.972970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5541.793315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.688150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.502586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.357820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.161078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.133163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.909174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.772496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.713275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.515119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.531810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.488231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.432483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.269115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.018531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.813125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.731705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.549678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.364425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.064593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.882725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.882296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.827618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.878887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.741653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.669454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.521869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.414480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.138561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.981887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.825421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.770458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.565384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.502138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.376545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.297873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.424951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.466370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.436337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.444445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.356285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.187843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.012824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.887884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.698222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.517808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.383295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.238440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.258920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.296151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.155160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.154521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.018362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.832211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.689790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.793787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.724077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.616214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.497088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.418427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.246906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.258938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.154426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.165119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.022674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.005230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.896585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.761448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.574814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.350476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.119107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.928914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.838340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.752285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.534835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.297184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.011820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.754871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.436569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.273838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.033810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.763519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.576544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.419166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.190229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.049612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.876337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.707692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.413661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.190434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.061820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.865865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.823396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.698302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.453592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.294912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.246400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.119341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.037648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.919369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.743571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.746187] LustreError: 172021:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5629.755746] LustreError: 172021:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5629.761560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.721883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.684665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.591731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.471296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.637239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.611509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.685067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.524819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.540895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.584897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.509051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.498900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.433611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.444536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.402441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.375068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.296531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.342160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.515570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.428124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.316303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.221336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.047081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.967542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.934522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.809417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.630354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.523377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.397624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.183322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.951174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.870471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.748309] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5660.761207] Lustre: Skipped 277 previous similar messages [ 5660.786458] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5660.794205] Lustre: Skipped 277 previous similar messages [ 5660.799918] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5660.806577] Lustre: Skipped 277 previous similar messages [ 5660.809825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.819447] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5660.827963] Lustre: Skipped 277 previous similar messages [ 5661.602594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.498371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.439712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.323459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.192122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.108614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.096558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.926760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.876405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.673767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.433734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.376606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.164604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.918734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.700741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.542218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.336944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.275333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.089084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.170452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.033876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.829118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.582346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.488157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.397158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.406845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.227723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.162687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.020111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.032886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.902109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.693767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.618034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.600472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.432383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.314051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.226741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.158646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.146866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.164620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.389419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.218045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.076968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.904972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.728761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.478499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.383133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.274220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.200177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.179860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.143657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.019198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.831498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.954399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.007516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.085049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.117386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.069119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.036665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.965369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.840805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.768199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.709515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.681124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.960589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.046604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.954713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.195981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.350974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.340086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.108658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.136564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.216395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.712545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.792131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.727590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.774336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.811542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.125139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.679461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.766827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.727746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.082089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.373342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.319959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.178385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.198257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.707135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.038207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.087360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.536955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.943956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.110657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.672968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.087195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.137788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.421123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.716610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.181239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.436633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.315478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.790596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.962883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.821062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.825806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.726140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.595294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.482543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.472289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.389523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.502195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.332258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.291059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.122382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.103686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.017165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.791615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.563894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.433353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.425045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.413424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.421154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.427825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.611549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.552309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.592869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.526770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.409871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.348367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.538380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.566943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.506653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.634330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.594316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.537208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.447349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.267456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.121717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.103695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.992569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.065554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.126649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.297368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.159628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.164618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.187351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.229776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.110614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.101057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.947381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.763069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.634664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.811329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.070544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.235108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.163323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.236552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.090390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.302460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.295511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.559265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.443685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.335310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.709260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.577117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.678387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.528025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.392039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.351942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.325404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.260092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.281923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.305507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.207130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.062301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.007288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.914232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.708214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.523450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.322588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.161454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.053882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.015728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.866563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.638322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.530674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.418506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.284374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.157793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.987562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.803450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.801938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.610855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.540093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.368346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.406458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.303495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.298464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.275327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.185406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.180804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.198504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.040875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.095941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.403500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.931809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.279367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.698102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.958556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.967023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.307185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.627659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.513940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.637228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.718795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.951833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.242560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.528692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.722295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.904165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.062193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.246879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.138484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.150262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.046257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.926568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.823613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.705762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.431700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.333379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.069121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.892713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.717931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.607269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.426916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.207162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.065930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.091523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.079685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.165911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.980311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.811173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.632205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.563363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.370970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.234585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.957520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.757526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.508695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.359039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.166748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.010183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.759701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.491514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.298670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.110709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.949359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.777286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.784241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.719275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.641539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.426473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.169893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.950110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.851107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.643795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.553618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.525520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.396278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.421221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.380727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.284883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.131523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.911492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.757954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.572859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.484312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.485449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.275093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.442370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.469405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.443624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.272241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.202361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.245312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.121766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.045931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.993598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.018098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.006337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.966174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.204913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.407368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.279879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.181573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.077944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.138512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.139338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.082325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.902250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.914166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.902133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.846334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.671061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.607995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.537177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.404753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.156587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.915286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.730429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.628233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.434575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.364379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.173367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.045692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.951370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.856500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.607988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.349924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.243361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.247358] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.250468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.105982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.401336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.294964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.334662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.365581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.465240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.333408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.416386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.631542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.686872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.933921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.993780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.942204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.951898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.963150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.388356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.334847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.351810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.184312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.311223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.378543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.469164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.306494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.585524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.759160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.559810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.585873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.829707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.745674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.188910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.543637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.827624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.950254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.977484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.056388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.921685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.950560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.839927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.824778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.788636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.908058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.870910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.875546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.684037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.542279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.218789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.103696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.196835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.164343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.196596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.970469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.966857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.781183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.064613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.177841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.377763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.426310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.593331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.718097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.600799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.596695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6037.578506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.590805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.804637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.671918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.819660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.593961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.596788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.871342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.017721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.976253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.036561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.015972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.068080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.189691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.490310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.667040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.520835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.689602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.629604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.721856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.565879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.636567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.653739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.694550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.694729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.657980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.432294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.385905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.321578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.303905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.176799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.160111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.933945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.732446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.522533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.483271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.265318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.167320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.006611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.938619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.048936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.751239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.812086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.820378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.781600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.718709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.784458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.112522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.249133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.412460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.599346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.462752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.537312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.606768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.464882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.586686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.705662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.903705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.258945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.844549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.476482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.741380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.731865] LustreError: 186397:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6499.737571] LustreError: 172021:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7165.420785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7165.431902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7165.451455] Lustre: Skipped 445 previous similar messages [ 7165.458491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.028758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7169.030972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.048984] LustreError: Skipped 2 previous similar messages [ 7171.405886] Lustre: server umount lustre-OST0000 complete [ 7177.184379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7177.201139] LustreError: Skipped 2 previous similar messages [ 7177.232462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7177.245598] Lustre: Skipped 1 previous similar message [ 7179.744770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7179.763863] Lustre: Skipped 1 previous similar message [ 7183.088066] Lustre: server umount lustre-MDT0000 complete [ 7184.872159] LustreError: 172729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.912610] LustreError: 172729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7188.466483] LustreError: 171981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445425 with bad export cookie 17631583984388332712 [ 7188.470905] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7188.485181] LustreError: 171981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7188.807589] Lustre: server umount lustre-MDT0001 complete [ 7204.403729] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 7207.411085] Key type lgssc unregistered [ 7207.803789] LNet: 227272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7207.811394] LNetError: 227272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7207.826200] LNet: Removed LNI 192.168.206.143@tcp [ 7208.861222] Key type .llcrypt unregistered [ 7208.864350] Key type ._llcrypt unregistered [ 7235.986122] Key type ._llcrypt registered [ 7235.988212] Key type .llcrypt registered [ 7236.096773] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7237.491080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7237.524881] alg: No test for adler32 (adler32-zlib) [ 7238.614174] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7238.967325] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 7240.679158] Key type lgssc registered [ 7241.897551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7250.257938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7250.282494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7251.758184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.266736] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7261.158679] LustreError: 228687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7261.180965] LustreError: 228687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7264.651169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7264.987985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7269.386947] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7277.594305] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7284.083931] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7292.064134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7292.615696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7296.761144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7300.007274] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7309.689718] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7316.046600] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7318.074901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.083066] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7318.092850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7318.115166] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7318.123140] LustreError: 228706:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7322.725199] Lustre: lustre-OST0000: Not available for connect from 192.168.206.43@tcp (stopping) [ 7325.668431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.678747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7325.710969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.735221] Lustre: Skipped 1 previous similar message [ 7326.826180] Lustre: server umount lustre-OST0000 complete [ 7338.175185] LustreError: 228706:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7362.467705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7362.636388] LustreError: 231897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7362.934741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7366.023244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7366.037402] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7366.038443] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7369.548654] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7377.876480] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7379.842609] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7386.646069] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92f3119e2000.ost_server_uuid 50 [ 7388.369891] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92f3119e2000.ost_server_uuid in FULL state after 0 sec [ 7393.761213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7393.776867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.797721] Lustre: Skipped 1 previous similar message [ 7393.806821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7396.833822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7396.837601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7396.845746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.854605] Lustre: Skipped 1 previous similar message [ 7399.806962] Lustre: server umount lustre-OST0000 complete [ 7407.076075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.097791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7407.105170] Lustre: Skipped 1 previous similar message [ 7410.899916] Lustre: server umount lustre-MDT0000 complete [ 7412.193669] LustreError: 231544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7412.218581] LustreError: 231544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7414.779841] LustreError: 228667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445651 with bad export cookie 7964577153402348929 [ 7414.792403] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7414.798924] LustreError: 228667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7415.095655] Lustre: server umount lustre-MDT0001 complete [ 7427.395837] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 7430.090958] Key type lgssc unregistered [ 7430.414869] LNet: 233897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7430.438593] LNetError: 233897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7430.459154] LNet: Removed LNI 192.168.206.143@tcp [ 7431.611364] Key type .llcrypt unregistered [ 7431.615392] Key type ._llcrypt unregistered [ 7466.916241] Key type ._llcrypt registered [ 7466.919474] Key type .llcrypt registered [ 7467.029380] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7467.766971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7467.825382] alg: No test for adler32 (adler32-zlib) [ 7468.899270] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7469.196419] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 7470.896779] Key type lgssc registered [ 7472.342732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7482.118965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7482.148767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.716477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.563106] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7494.125182] LustreError: 235627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7494.155618] LustreError: 235627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7498.305035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7498.582273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7504.165837] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7507.683438] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7515.501142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7516.172944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7522.509216] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7529.515836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7531.439997] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7531.636564] Lustre: lustre-OST0001: new disk, initializing [ 7531.645811] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7531.658434] Lustre: 237936:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7531.823957] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7532.272166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7532.282913] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7532.425779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7539.369087] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7552.844067] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7562.568747] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7578.594894] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.604744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7578.613782] Lustre: Skipped 2 previous similar messages [ 7578.628524] Lustre: Skipped 3 previous similar messages [ 7583.713824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7583.725116] Lustre: Skipped 3 previous similar messages [ 7588.847186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7591.391138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7591.547160] Lustre: server umount lustre-MDT0000 complete [ 7593.968375] LustreError: 236355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7593.995643] LustreError: 236355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7595.131454] LustreError: 235607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445832 with bad export cookie 9923428237031674364 [ 7595.140793] LustreError: 235607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7595.142617] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7595.389050] Lustre: server umount lustre-MDT0001 complete [ 7608.447830] Lustre: server umount lustre-OST0000 complete [ 7622.192556] Lustre: server umount lustre-OST0001 complete [ 7643.878422] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7653.612566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7653.887469] LustreError: 241046:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7653.948866] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7658.665487] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7658.987403] LustreError: 241047:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7663.141506] LustreError: 241046:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7668.414814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.886253] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7672.915889] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7675.713533] Lustre: 242189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7681.968171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7689.262471] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7693.628777] LustreError: 242545:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.646602] LustreError: 242545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7698.296129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7698.518272] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7698.524869] Lustre: Skipped 1 previous similar message [ 7699.590389] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49760) [ 7704.437462] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7711.990538] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7715.997028] Lustre: 244057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7730.151321] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7730.155358] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7730.169532] Lustre: Skipped 1 previous similar message [ 7730.175140] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7730.185996] Lustre: Skipped 3 previous similar messages [ 7734.870975] Lustre: server umount mylustre-MDT0000 complete [ 7735.793151] LustreError: 242120:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7735.816145] LustreError: 242120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7739.190235] LustreError: 244259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445976 with bad export cookie 9923428237031677171 [ 7739.190488] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7739.200060] LustreError: 244259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7739.504084] Lustre: server umount mylustre-MDT0001 complete [ 7753.435331] Lustre: server umount mylustre-OST0000 complete [ 7757.033179] Lustre: 234890:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445977/real 1788445977] req@ffff89b906859180 x1875321244774656/t0(0) o400->mylustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788445993 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7757.088396] Lustre: mylustre-MDT0001-lwp-OST0001: Connection to mylustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7757.107877] Lustre: Skipped 3 previous similar messages [ 7757.831849] Lustre: server umount mylustre-OST0001 complete [ 7780.266189] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7790.993709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7791.343257] LustreError: 246368:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.419825] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.240962] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7805.107133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7809.454216] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7811.951917] Lustre: 247509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7818.006285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7824.336099] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7832.462272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7836.877384] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49792) [ 7838.623334] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7846.554888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7850.686461] Lustre: 249381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7857.646383] Lustre: tfs-MDT0000-lwp-OST0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7857.655960] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7857.660406] Lustre: Skipped 2 previous similar messages [ 7857.675234] Lustre: Skipped 5 previous similar messages [ 7863.217127] Lustre: server umount tfs-MDT0000 complete [ 7866.715962] LustreError: 246343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446103 with bad export cookie 9923428237031678837 [ 7866.731676] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7867.025524] Lustre: server umount tfs-MDT0001 complete [ 7880.798944] Lustre: server umount tfs-OST0000 complete [ 7893.340665] Lustre: server umount tfs-OST0001 complete [ 7913.142132] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 7923.345513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7923.748190] LustreError: 251631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.780986] LustreError: 251631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7923.890412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7923.900985] Lustre: Skipped 3 previous similar messages [ 7928.731923] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7936.996118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7941.654935] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7944.892877] Lustre: 252772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7951.713164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7957.750874] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7967.371404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7968.713141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49762 to 0x280000401:49824) [ 7975.042610] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 7983.311770] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7987.801306] Lustre: 254645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7996.357395] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 10:37:11 (1788446231) [ 8014.307632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8014.325061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8014.338055] Lustre: Skipped 1 previous similar message [ 8014.346498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8014.350527] Lustre: Skipped 4 previous similar messages [ 8018.817935] Lustre: server umount lustre-MDT0000 complete [ 8022.112212] LustreError: 252774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446258 with bad export cookie 9923428237031680279 [ 8022.117302] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8022.126389] LustreError: 252774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8025.064684] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8025.087581] Lustre: Skipped 4 previous similar messages [ 8028.546600] Lustre: server umount lustre-MDT0001 complete [ 8038.356755] Lustre: server umount lustre-OST0000 complete [ 8056.287114] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8056.506900] Lustre: server umount lustre-OST0001 complete [ 8062.656150] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 8074.518323] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8108.671895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.882534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8108.906504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8108.964379] Lustre: lustre-MDT0000: new disk, initializing [ 8109.035560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8109.045227] Lustre: Skipped 3 previous similar messages [ 8109.057596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8113.142381] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8122.948119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8123.064951] Lustre: 257835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8123.132988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8123.139512] Lustre: Skipped 1 previous similar message [ 8123.231786] Lustre: lustre-MDT0001: new disk, initializing [ 8123.436558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8123.454173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8127.576539] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8137.585802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8137.901091] Lustre: lustre-OST0000: new disk, initializing [ 8137.911209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8137.919546] Lustre: 259417:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8139.517175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8139.535584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8139.659438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8144.925304] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8155.262250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8155.504134] Lustre: lustre-OST0001: new disk, initializing [ 8155.508963] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8155.516924] Lustre: 260287:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8157.194872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8157.210535] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8157.327499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8162.030123] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8175.439633] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 10:40:10 (1788446410) [ 8192.267891] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 10:40:27 (1788446427) [ 8208.354754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.363423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8208.370033] Lustre: Skipped 1 previous similar message [ 8208.376047] Lustre: Skipped 8 previous similar messages [ 8221.151127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8221.283306] Lustre: server umount lustre-OST0000 complete [ 8223.719698] LustreError: 259515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8223.749428] LustreError: 259515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8225.767380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8231.249941] Lustre: server umount lustre-MDT0000 complete [ 8234.736645] LustreError: 258629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446471 with bad export cookie 9923428237031681609 [ 8234.739801] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8234.748936] LustreError: 258629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8244.193572] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8244.197791] Lustre: Skipped 9 previous similar messages [ 8249.312145] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8249.510296] Lustre: server umount lustre-MDT0001 complete [ 8260.253175] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 8333.614398] Lustre: server umount lustre-OST0001 complete [ 8340.068298] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 8355.208657] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8390.557324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.748693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8390.767802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8390.827636] Lustre: lustre-MDT0000: new disk, initializing [ 8390.882307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8390.888656] Lustre: Skipped 3 previous similar messages [ 8390.902594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8394.872178] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8403.592031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8403.684315] Lustre: 265145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8403.873361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8407.554504] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8416.002373] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8421.425345] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8428.334682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8428.598169] Lustre: lustre-OST0000: new disk, initializing [ 8428.600591] Lustre: Skipped 1 previous similar message [ 8428.604947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8428.608862] Lustre: Skipped 2 previous similar messages [ 8428.613727] Lustre: 267118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8430.213402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8430.227362] Lustre: Skipped 1 previous similar message [ 8430.229969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8430.355392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8435.217597] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8444.234750] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8456.164735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.167383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8456.187876] Lustre: Skipped 5 previous similar messages [ 8456.200701] Lustre: Skipped 2 previous similar messages [ 8457.774357] Lustre: server umount lustre-OST0000 complete [ 8467.528785] Lustre: server umount lustre-MDT0000 complete [ 8471.316281] LustreError: 265137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446708 with bad export cookie 9923428237031683072 [ 8471.317632] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8471.327471] LustreError: 265137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8471.637378] Lustre: server umount lustre-MDT0001 complete [ 8483.436903] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 8486.150373] Key type lgssc unregistered [ 8486.407607] LNet: 268876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8486.421502] LNetError: 268876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8486.441479] LNet: Removed LNI 192.168.206.143@tcp [ 8487.295359] Key type .llcrypt unregistered [ 8487.301302] Key type ._llcrypt unregistered [ 8513.728914] Key type ._llcrypt registered [ 8513.734311] Key type .llcrypt registered [ 8513.828760] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8514.884317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8514.951189] alg: No test for adler32 (adler32-zlib) [ 8516.013644] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8516.247812] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 8517.951230] Key type lgssc registered [ 8518.925073] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8525.962276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8525.985429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.421884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8531.391867] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8537.585357] LustreError: 270333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.612991] LustreError: 270333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8538.984894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8539.374153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8543.367180] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8549.996678] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8555.690306] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8561.420083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8561.751778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8567.801610] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8573.050641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8574.797052] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8575.810426] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8578.541314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8578.544519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8578.554395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8578.562141] Lustre: Skipped 1 previous similar message [ 8579.717819] Lustre: server umount lustre-OST0000 complete [ 8583.650382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8583.659044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8583.664630] Lustre: Skipped 2 previous similar messages [ 8583.681562] Lustre: Skipped 1 previous similar message [ 8588.782444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8588.788046] Lustre: Skipped 1 previous similar message [ 8589.796540] Lustre: server umount lustre-MDT0000 complete [ 8593.893374] LustreError: 271063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8593.909281] LustreError: 271063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8594.400146] LustreError: 272867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446831 with bad export cookie 12008554247403415503 [ 8594.406759] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8594.411328] LustreError: 272867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8595.060095] Lustre: server umount lustre-MDT0001 complete [ 8607.297314] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 8609.706716] Key type lgssc unregistered [ 8609.959310] LNet: 273921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8609.969601] LNetError: 273921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8610.985136] LNet: Removed LNI 192.168.206.143@tcp [ 8611.702921] Key type .llcrypt unregistered [ 8611.705679] Key type ._llcrypt unregistered [ 8627.772250] Key type ._llcrypt registered [ 8627.774982] Key type .llcrypt registered [ 8627.888112] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8629.042539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8629.157873] alg: No test for adler32 (adler32-zlib) [ 8630.192674] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8630.404044] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 8632.039189] Key type lgssc registered [ 8633.027601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8641.209683] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8641.232764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8656.799396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8661.919402] LustreError: 275112:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.143@tcp: failed processing log, type 4: rc = -110 [ 8688.607254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8693.817436] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8701.293267] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8703.713402] Lustre: server umount lustre-OST0000 complete [ 8720.241427] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 8722.814756] Key type lgssc unregistered [ 8723.234942] LNet: 276742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8723.249156] LNetError: 276742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8723.284058] LNet: Removed LNI 192.168.206.143@tcp [ 8724.417231] Key type .llcrypt unregistered [ 8724.420219] Key type ._llcrypt unregistered [ 8760.047346] Key type ._llcrypt registered [ 8760.050166] Key type .llcrypt registered [ 8760.162553] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8761.526061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8761.547268] alg: No test for adler32 (adler32-zlib) [ 8762.693209] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8763.092773] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 8764.759176] Key type lgssc registered [ 8766.060987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8778.936356] loop0: detected capacity change from 0 to 204800000 [ 8779.091445] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8779.110669] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8779.332682] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8780.285088] loop0: detected capacity change from 0 to 204800000 [ 8780.460437] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8780.466551] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8780.770674] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8781.733358] loop0: detected capacity change from 0 to 204800000 [ 8781.934917] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8781.948959] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8782.056136] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8783.217643] loop0: detected capacity change from 0 to 204800000 [ 8783.464505] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8784.666667] loop0: detected capacity change from 0 to 204800000 [ 8784.704235] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8788.029449] loop0: detected capacity change from 0 to 204800000 [ 8788.102083] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8791.716271] loop0: detected capacity change from 0 to 204800000 [ 8791.742187] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8795.607068] loop0: detected capacity change from 0 to 204800000 [ 8795.637105] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8799.135875] loop0: detected capacity change from 0 to 204800000 [ 8799.208651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8799.243470] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.312363] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8799.319575] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8799.329527] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8799.338388] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8799.345545] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8799.352240] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8799.359930] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8799.366686] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8799.374270] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8799.382378] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8799.390090] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8799.402457] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8799.408913] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8799.414982] Lustre: 280152:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8804.944963] Lustre: server umount MGS complete [ 8805.616650] loop0: detected capacity change from 0 to 204800000 [ 8805.659269] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8806.074771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8806.133974] Lustre: lustre-MDT0000: Aborting client recovery [ 8807.150773] loop1: detected capacity change from 0 to 204800000 [ 8807.295933] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8807.396798] Lustre: 280583:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8807.413753] Lustre: 280583:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8807.436769] Lustre: 280583:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8807.454621] Lustre: 280583:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8807.473886] Lustre: 280583:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8807.487552] Lustre: 280583:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8807.499936] Lustre: 280583:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8807.533412] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8807.725573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8807.779669] Lustre: lustre-MDT0001: Aborting client recovery [ 8808.746189] loop2: detected capacity change from 0 to 204800000 [ 8808.866431] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8808.960074] Lustre: 280665:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8808.978837] Lustre: 280665:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8808.988466] Lustre: 280665:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8809.005414] Lustre: 280665:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8809.023448] Lustre: 280665:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8809.046964] Lustre: 280665:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8809.072940] Lustre: 280665:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8809.237722] Lustre: Found index 0 for lustre-OST0000, updating log [ 8809.269575] Lustre: 280684:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8809.333836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.352456] Lustre: lustre-OST0000: Aborting recovery [ 8810.385069] loop3: detected capacity change from 0 to 204800000 [ 8810.587522] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8810.667696] Lustre: 280749:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8810.688909] Lustre: 280749:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8810.699851] Lustre: 280749:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8810.718980] Lustre: 280749:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8810.732891] Lustre: 280749:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8810.745683] Lustre: 280749:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8810.756587] Lustre: 280749:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8810.777031] Lustre: Found index 1 for lustre-OST0001, updating log [ 8810.810682] Lustre: 280756:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8810.886857] Lustre: lustre-OST0001: Aborting recovery [ 8815.984225] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8815.989495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8816.052053] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8816.052751] LustreError: 280527:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8816.094971] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8818.037701] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8818.051124] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8820.291835] LustreError: 280494:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 10382us: rc = -2 [ 8820.889489] LustreError: 280495:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 408us: rc = -2 [ 8820.911096] LustreError: 280495:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 22 previous similar messages [ 8833.511375] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.514499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8833.529062] Lustre: Skipped 3 previous similar messages [ 8833.536725] Lustre: Skipped 2 previous similar messages [ 8833.551290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8838.627032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8838.642623] Lustre: Skipped 4 previous similar messages [ 8839.744792] Lustre: server umount lustre-MDT0000 complete [ 8840.544315] LustreError: 280477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447077 with bad export cookie 624663497004000864 [ 8840.545733] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8840.554099] LustreError: 280477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8843.747344] LustreError: 280493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8843.747672] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.748283] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8843.769066] LustreError: 280493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8843.786043] Lustre: Skipped 1 previous similar message [ 8845.792077] LustreError: 281016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8845.792980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8845.808598] Lustre: Skipped 2 previous similar messages [ 8846.937784] Lustre: server umount lustre-MDT0001 complete [ 8862.175164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8862.374235] Lustre: server umount lustre-OST0000 complete [ 8869.467209] Lustre: server umount lustre-OST0001 complete [ 8876.348963] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 10:51:51 (1788447111) [ 8903.318528] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 8912.376910] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 8946.262095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8946.524135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8946.575791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8946.654177] Lustre: lustre-MDT0000: new disk, initializing [ 8946.711687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8946.716988] Lustre: Skipped 1 previous similar message [ 8946.737609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8950.859294] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8961.329398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8961.485401] Lustre: 284823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8961.538946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8961.545498] Lustre: Skipped 1 previous similar message [ 8961.631315] Lustre: lustre-MDT0001: new disk, initializing [ 8961.763643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8961.833460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8961.855622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8966.690101] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8975.510434] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8981.179276] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8987.567307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8987.834060] Lustre: lustre-OST0000: new disk, initializing [ 8987.836701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8987.841284] Lustre: 286790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8987.938166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8994.355796] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 8996.970686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8996.991525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8997.148523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9003.676307] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9008.125396] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9017.057479] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9022.489990] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9074.150146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9074.151991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9074.158953] LustreError: Skipped 1 previous similar message [ 9074.160169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9074.193435] Lustre: Skipped 1 previous similar message [ 9075.460131] Lustre: server umount lustre-OST0000 complete [ 9079.273342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9079.290971] Lustre: Skipped 1 previous similar message [ 9084.387995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9084.400436] Lustre: Skipped 4 previous similar messages [ 9085.603215] Lustre: server umount lustre-MDT0000 complete [ 9089.504539] LustreError: 287461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9089.529414] LustreError: 287461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9089.882821] LustreError: 287992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447326 with bad export cookie 624663497004003573 [ 9089.896845] LustreError: 287992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9089.901600] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9090.190990] Lustre: server umount lustre-MDT0001 complete [ 9098.990374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9103.409910] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9108.180235] Lustre: server umount MGS complete [ 9119.537450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9120.012915] LustreError: 290608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9120.041971] LustreError: 290621:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9120.110204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9124.394566] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9125.351162] LustreError: 290609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.262371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9137.474551] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9144.827735] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9150.752461] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9158.275409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9158.610773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9158.618472] Lustre: Skipped 1 previous similar message [ 9165.770132] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9173.956937] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9200.609110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9200.625464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.651256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9202.255166] Lustre: server umount lustre-OST0000 complete [ 9208.812277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9208.820287] Lustre: Skipped 1 previous similar message [ 9212.253784] Lustre: server umount lustre-MDT0000 complete [ 9213.922778] LustreError: 290609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9213.947827] LustreError: 290609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9215.716942] LustreError: 293336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447452 with bad export cookie 624663497004005393 [ 9215.730320] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9225.311107] Lustre: server umount lustre-MDT0001 complete [ 9233.866904] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 9236.387307] Key type lgssc unregistered [ 9236.642749] LNet: 294553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9236.649752] LNetError: 294553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9236.669843] LNet: Removed LNI 192.168.206.143@tcp [ 9237.385824] Key type .llcrypt unregistered [ 9237.390904] Key type ._llcrypt unregistered [ 9273.004716] Key type ._llcrypt registered [ 9273.007627] Key type .llcrypt registered [ 9273.094525] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 9286.566971] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 9287.689557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9287.759507] alg: No test for adler32 (adler32-zlib) [ 9288.895662] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9289.166549] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 9290.815343] Key type lgssc registered [ 9292.140590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9327.829934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9327.881058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9329.147644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9329.186245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9329.277219] Lustre: lustre-MDT0000: new disk, initializing [ 9329.461258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9329.494459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9334.907697] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9346.107211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9346.273860] Lustre: 298506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9346.332407] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9346.336402] Lustre: Skipped 1 previous similar message [ 9346.452493] Lustre: lustre-MDT0001: new disk, initializing [ 9346.618657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9346.686619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9346.701944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9351.462962] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9360.245561] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9365.298686] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9372.240752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9372.540827] Lustre: lustre-OST0000: new disk, initializing [ 9372.547277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9372.557790] Lustre: 300477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9372.685654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9378.376497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9378.398310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9378.578947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9379.551828] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9388.791487] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9394.739835] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9403.847610] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9411.804141] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9421.741430] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9465.826408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9465.831357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9465.842636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9465.848134] Lustre: Skipped 1 previous similar message [ 9465.864661] Lustre: Skipped 1 previous similar message [ 9470.944856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9470.952965] Lustre: Skipped 1 previous similar message [ 9476.074227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9476.082095] Lustre: Skipped 1 previous similar message [ 9479.135141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9479.262334] Lustre: server umount lustre-OST0000 complete [ 9484.769112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9484.776339] LustreError: Skipped 1 previous similar message [ 9484.779602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9484.798733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9486.306704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.031716] Lustre: server umount lustre-MDT0000 complete [ 9492.494889] LustreError: 300476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447729 with bad export cookie 15570492706352138693 [ 9492.500916] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9492.509152] LustreError: 300476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9492.782641] Lustre: server umount lustre-MDT0001 complete [ 9501.123942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9505.346829] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9509.758204] Lustre: server umount MGS complete [ 9520.643424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9521.048537] LustreError: 304359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9521.077079] LustreError: 304359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9521.099937] LustreError: 304372:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9521.170896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9525.403604] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9526.242469] LustreError: 304360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9530.346795] LustreError: 304359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9535.248653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9535.459224] LustreError: 304360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9535.748378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9541.112776] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9548.683510] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9554.281968] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9561.446392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9561.836753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9570.089665] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9578.318780] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9611.235855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9611.237314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9611.241410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9611.241421] Lustre: Skipped 2 previous similar messages [ 9611.276294] Lustre: Skipped 1 previous similar message [ 9615.544977] Lustre: server umount lustre-OST0000 complete [ 9621.475595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9621.492123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9621.502649] Lustre: Skipped 2 previous similar messages [ 9625.484221] Lustre: server umount lustre-MDT0000 complete [ 9626.592664] LustreError: 305919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9626.617877] LustreError: 305919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9629.202325] LustreError: 304342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447866 with bad export cookie 15570492706352140646 [ 9629.202599] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9629.223678] LustreError: 304342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9629.595946] Lustre: server umount lustre-MDT0001 complete [ 9639.843983] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [ 9642.479461] Key type lgssc unregistered [ 9642.749679] LNet: 308306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9642.762090] LNetError: 308306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9642.779898] LNet: Removed LNI 192.168.206.143@tcp [ 9643.713921] Key type .llcrypt unregistered [ 9643.716953] Key type ._llcrypt unregistered [ 9665.166346] Key type ._llcrypt registered [ 9665.170921] Key type .llcrypt registered [ 9665.261208] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [ 9678.986115] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 9679.761875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9679.784990] alg: No test for adler32 (adler32-zlib) [ 9680.785888] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9681.002808] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [ 9682.655290] Key type lgssc registered [ 9683.698034] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9709.514226] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [ 9724.018557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9724.055590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9725.347184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9725.398707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9725.487741] Lustre: lustre-MDT0000: new disk, initializing [ 9725.610487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9725.629535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9730.095664] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9741.675246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9742.074034] Lustre: lustre-OST0000: new disk, initializing [ 9742.079306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9742.085973] Lustre: Skipped 1 previous similar message [ 9742.092301] Lustre: 312354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9742.183474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9749.623087] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [ 9751.129232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9751.137867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9751.172179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10652.136281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10652.153633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10655.261929] Lustre: server umount lustre-MDT0000 complete [10659.152773] LustreError: 313342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448896 with bad export cookie 6775747779321419523 [10659.153855] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10659.161438] LustreError: 313342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10659.315234] Lustre: server umount lustre-OST0000 complete [10684.848713] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [10696.882845] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [10737.215277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10737.469631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10737.489112] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10737.573745] Lustre: lustre-MDT0000: new disk, initializing [10737.644354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10737.665912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10741.981262] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10750.931642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10751.112958] Lustre: 316700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10751.158838] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10751.162315] Lustre: Skipped 1 previous similar message [10751.221493] Lustre: lustre-MDT0001: new disk, initializing [10751.317753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10751.364551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10751.375091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10755.844229] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10764.648787] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10769.807267] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10776.341769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10776.674740] Lustre: lustre-OST0000: new disk, initializing [10776.684935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10776.694274] Lustre: 318667:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10776.812303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10782.929660] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10785.344038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10785.356612] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10785.450812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10791.996346] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10796.914088] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10797.124910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10801.666938] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10801.893371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10806.240669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10806.253590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10806.266365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10806.267675] Lustre: Skipped 1 previous similar message [10806.281797] Lustre: Skipped 1 previous similar message [10811.365780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10811.373420] Lustre: Skipped 1 previous similar message [10816.481513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10816.494126] Lustre: Skipped 1 previous similar message [10818.015186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10818.314923] Lustre: server umount lustre-OST0000 complete [10823.137519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10823.142721] LustreError: Skipped 1 previous similar message [10823.145544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10823.157857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10826.721352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10828.659597] Lustre: server umount lustre-MDT0000 complete [10832.617229] LustreError: 316692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449069 with bad export cookie 6775747779324360335 [10832.622695] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10832.634850] LustreError: 316692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10833.172404] Lustre: server umount lustre-MDT0001 complete [10837.139044] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10839.190511] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 11:24:34 (1788449074) [10845.920346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10846.358879] LustreError: 320983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.376887] LustreError: 320983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10846.449697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10851.060124] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10851.812178] LustreError: 320984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.929149] LustreError: 320983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10860.377944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10860.867208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10865.606485] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10873.410888] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10879.476746] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10887.145342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10887.564863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10895.065735] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10903.547851] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10911.566272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10911.743721] Lustre: lustre-OST0001: new disk, initializing [10911.749582] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10911.764037] Lustre: 323818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10911.893055] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10918.634826] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [10922.089986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10922.105266] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10922.158431] LustreError: 324299:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10923.245619] LustreError: 324303:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10923.254592] LustreError: 324303:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10924.266680] LustreError: 324299:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10926.376415] LustreError: 324299:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10926.385657] LustreError: 324299:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [10928.896270] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10929.050137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10930.534720] LustreError: 324299:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10930.560333] LustreError: 324299:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 8 previous similar messages [10934.489192] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92f3119e2800.ost_server_uuid 50 [10935.810099] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92f3119e2800.ost_server_uuid in FULL state after 0 sec [10938.795869] LustreError: 324303:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10938.805310] LustreError: 324303:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [10941.020994] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92f3119e2800.ost_server_uuid 50 [10942.509314] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92f3119e2800.ost_server_uuid in FULL state after 0 sec [10944.003545] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10958.324055] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.329838] Lustre: Skipped 1 previous similar message [10958.329861] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10958.345993] Lustre: Skipped 3 previous similar messages [10963.435293] LustreError: 322803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.449974] LustreError: 322803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10963.531907] Lustre: server umount lustre-OST0001 complete [10968.551406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10968.556372] LustreError: 324586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10968.561322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10968.587853] LustreError: 324586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10974.790168] Lustre: server umount lustre-OST0000 complete [10978.786095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10978.798238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10978.802393] Lustre: Skipped 6 previous similar messages [10984.746668] Lustre: server umount lustre-MDT0000 complete [10988.544494] LustreError: 320965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449225 with bad export cookie 6775747779324361217 [10988.547804] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10988.553300] LustreError: 320965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10988.905771] Lustre: server umount lustre-MDT0001 complete [10998.796765] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [11001.316658] Key type lgssc unregistered [11001.578931] LNet: 326507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11001.589703] LNetError: 326507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11001.607046] LNet: Removed LNI 192.168.206.143@tcp [11002.335715] Key type .llcrypt unregistered [11002.339302] Key type ._llcrypt unregistered [11024.173255] Key type ._llcrypt registered [11024.175012] Key type .llcrypt registered [11024.353033] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [11025.230544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11025.262467] alg: No test for adler32 (adler32-zlib) [11026.447930] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11026.707631] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [11028.351328] Key type lgssc registered [11029.563472] Lustre: Echo OBD driver; http://www.lustre.org/ [11036.496901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11036.525048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11037.932731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11042.180893] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11047.397117] LustreError: 327920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11047.439974] LustreError: 327920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11049.146977] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11052.523051] LustreError: 327919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.758822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.283657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11063.246878] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11074.039560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11074.485833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11081.010586] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11082.667549] LustreError: 329600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11082.684533] LustreError: 329600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11082.686352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11087.420811] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11087.849534] LustreError: 329600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.863392] LustreError: 329600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11092.987623] LustreError: 330072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11093.002728] LustreError: 330072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11094.092278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11094.305923] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11099.635087] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11099.736782] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11105.885420] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11112.328669] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11112.483846] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11222.505116] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11222.508289] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11222.514560] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11222.523381] Lustre: Skipped 1 previous similar message [11222.545724] Lustre: Skipped 1 previous similar message [11227.634949] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11227.647811] Lustre: Skipped 1 previous similar message [11232.223272] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11232.335474] Lustre: server umount lustre-OST0001 complete [11232.754994] LustreError: 330072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11232.782376] LustreError: 330072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11241.440214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11241.448246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11241.448741] LustreError: Skipped 2 previous similar messages [11241.466156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11248.097724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11248.101784] Lustre: Skipped 3 previous similar messages [11253.215160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11253.229805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11253.230055] LustreError: 330072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11253.230062] LustreError: 330072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11253.244722] Lustre: Skipped 2 previous similar messages [11253.357073] Lustre: server umount lustre-OST0000 complete [11261.920149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11261.930371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.947312] Lustre: Skipped 1 previous similar message [11261.969080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11264.784319] Lustre: server umount lustre-MDT0000 complete [11268.698624] LustreError: 331605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449505 with bad export cookie 3413647940308376200 [11268.706738] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11268.722085] LustreError: 331605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11269.037793] Lustre: server umount lustre-MDT0001 complete [11280.070652] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [11282.477861] Key type lgssc unregistered [11282.819891] LNet: 332864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11282.834844] LNetError: 332864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11282.858093] LNet: Removed LNI 192.168.206.143@tcp [11283.682107] Key type .llcrypt unregistered [11283.685818] Key type ._llcrypt unregistered [11327.630477] Key type ._llcrypt registered [11327.633152] Key type .llcrypt registered [11327.727683] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [11328.845750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11329.034420] alg: No test for adler32 (adler32-zlib) [11330.139891] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11330.410293] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [11332.039170] Key type lgssc registered [11333.119205] Lustre: Echo OBD driver; http://www.lustre.org/ [11342.501454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11342.520989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11343.988228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11349.037667] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11353.577454] LustreError: 335151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11353.597671] LustreError: 335151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11357.789051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11358.281171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11363.703675] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11366.684172] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11375.677772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11375.967395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11383.150580] LustreError: 336642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11384.185734] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11388.260969] LustreError: 336645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11388.290732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11388.300861] LustreError: 336645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11393.522483] LustreError: 336644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11393.539599] LustreError: 336644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11393.594118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11393.789251] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11399.168934] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11399.309295] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11409.062841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11414.348439] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11429.865422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11429.867672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11429.867776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11429.878966] Lustre: Skipped 3 previous similar messages [11429.882580] Lustre: Skipped 2 previous similar messages [11431.399461] Lustre: server umount lustre-MDT0000 complete [11434.677800] LustreError: 335132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449671 with bad export cookie 9604051677088860341 [11434.679654] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11434.691104] LustreError: 335132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11434.976781] LustreError: 335146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11434.980885] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11435.007160] LustreError: 335146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11435.015695] Lustre: Skipped 1 previous similar message [11435.215289] Lustre: server umount lustre-MDT0001 complete [11441.025838] Lustre: server umount lustre-OST0000 complete [11444.993387] Lustre: server umount lustre-OST0001 complete [11462.853193] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [11473.055603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11473.342833] LustreError: 340414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11473.404224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11477.528714] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11485.447856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11485.951069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11490.535750] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11493.527197] Lustre: 341554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11500.020892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11506.507195] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11510.759319] LustreError: 341909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11510.786519] LustreError: 341909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11514.912107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11515.176789] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11515.184891] Lustre: Skipped 1 previous similar message [11518.272425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11518.288288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11522.747965] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11531.929411] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11536.348825] Lustre: 343427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11541.984240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11541.995926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11542.017374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11542.030526] Lustre: Skipped 1 previous similar message [11544.034397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11544.049504] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.069827] Lustre: Skipped 1 previous similar message [11547.034716] Lustre: server umount lustre-MDT0000 complete [11549.165200] LustreError: 340409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11549.191257] LustreError: 340409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11551.108730] LustreError: 340395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449787 with bad export cookie 9604051677088862035 [11551.111353] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11551.127558] LustreError: 340395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11551.383560] Lustre: server umount lustre-MDT0001 complete [11565.868056] Lustre: server umount lustre-OST0000 complete [11579.827581] Lustre: server umount lustre-OST0001 complete [11583.008302] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11584.597708] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11586.330935] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 11:37:01 (1788449821) [11607.085554] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [11615.989727] loop0: detected capacity change from 0 to 1099511627776 [11616.528978] SGI XFS with ACLs, security attributes, quota, no debug enabled [11616.541978] *** VALIDATE xfs *** [11616.548970] XFS (loop0): Mounting V5 Filesystem [11616.567665] XFS (loop0): Ending clean mount [11617.782439] loop1: detected capacity change from 0 to 18691697672192 [11734.611179] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [11745.456772] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [11787.900905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11788.196834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11788.230023] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11788.332601] Lustre: lustre-MDT0000: new disk, initializing [11788.447591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11788.473499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11793.222841] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11803.204095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11803.435950] Lustre: 348563:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11803.537603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11803.545949] Lustre: Skipped 1 previous similar message [11803.697107] Lustre: lustre-MDT0001: new disk, initializing [11803.922164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11803.930199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11808.887283] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11819.299853] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11824.539612] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11830.639919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11830.876854] Lustre: lustre-OST0000: new disk, initializing [11830.880181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11830.884820] Lustre: 350534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11832.587755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11832.600212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11832.698791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11836.902835] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11846.552841] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11851.135800] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11851.322944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11856.438462] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11856.721716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11863.520452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11863.530620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.547047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11863.556396] Lustre: Skipped 4 previous similar messages [11868.643092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11868.649695] Lustre: Skipped 1 previous similar message [11873.247205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11873.434909] Lustre: server umount lustre-OST0000 complete [11878.882467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11878.884334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11878.899873] Lustre: Skipped 2 previous similar messages [11878.915218] Lustre: Skipped 1 previous similar message [11883.580801] Lustre: server umount lustre-MDT0000 complete [11884.001928] LustreError: 349431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.022754] LustreError: 349431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11887.398756] LustreError: 348557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450124 with bad export cookie 9604051677088863589 [11887.411133] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11887.422173] LustreError: 348557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11887.811791] Lustre: server umount lustre-MDT0001 complete [11896.782337] XFS (loop0): Unmounting Filesystem [11901.546854] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 11:42:16 (1788450136) [11908.063077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11908.566419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11908.574061] Lustre: Skipped 2 previous similar messages [11913.179756] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11922.756551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11927.468968] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11934.431681] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11939.761837] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11946.024821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11951.350905] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [11958.135612] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11967.968723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11967.990747] LustreError: Skipped 1 previous similar message [11968.006300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11968.033192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11973.750248] Lustre: server umount lustre-OST0000 complete [11985.480465] Lustre: server umount lustre-MDT0000 complete [11989.966576] LustreError: 353076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450226 with bad export cookie 9604051677088864478 [11989.977463] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11990.313925] Lustre: server umount lustre-MDT0001 complete [12001.072805] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [12003.888995] Key type lgssc unregistered [12004.146106] LNet: 356722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12004.153982] LNetError: 356722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12004.169631] LNet: Removed LNI 192.168.206.143@tcp [12005.215321] Key type .llcrypt unregistered [12005.217308] Key type ._llcrypt unregistered [12033.617758] Key type ._llcrypt registered [12033.620178] Key type .llcrypt registered [12033.733682] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [12034.836763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12034.869690] alg: No test for adler32 (adler32-zlib) [12036.060299] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12036.298433] LNet: Added LNI 192.168.206.143@tcp [8/256/0/180] [12037.944234] Key type lgssc registered [12039.027807] Lustre: Echo OBD driver; http://www.lustre.org/ [12047.164427] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12047.183511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12048.674092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12054.426502] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12059.125021] LustreError: 358286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.151479] LustreError: 358286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12064.224360] LustreError: 358285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12064.903872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12065.535307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12070.414592] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12078.631444] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12085.339797] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12093.110223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12093.554152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12099.578369] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12107.062337] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12120.153150] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.43@tcp (stopping) [12121.575354] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12121.596810] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12121.601147] Lustre: Skipped 1 previous similar message [12123.878641] Lustre: server umount lustre-MDT0001 complete [12125.243944] LustreError: 358281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12130.370987] LustreError: 358281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12130.394413] LustreError: 358281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12134.098334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12134.252420] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12134.259197] Lustre: Found index 1 for lustre-MDT0001, updating log [12134.289458] Lustre: 358275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12134.579593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12138.863259] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12140.018965] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12141.043799] LustreError: 361844:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12141.049834] LustreError: 361844:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12141.058750] LustreError: 358277:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12141.135961] LustreError: 361852:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.143@tcp: cfg command failed: rc = -17 [12141.146655] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12141.146655] [12150.242928] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12150.243969] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12150.256153] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12150.263440] Lustre: Skipped 2 previous similar messages [12153.682813] Lustre: server umount lustre-MDT0001 complete [12155.367782] LustreError: 358281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12155.391556] LustreError: 358281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12163.874675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12163.986851] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12163.990457] Lustre: Found index 1 for lustre-MDT0001, updating log [12164.069812] Lustre: 358275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12164.415881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12168.118508] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12169.722925] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12173.332095] LustreError: 362971:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12173.339499] LustreError: 362971:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12173.346651] LustreError: 358277:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12173.355906] LustreError: 358277:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12173.432306] LustreError: 362977:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12173.436811] LustreError: 362977:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12173.440587] LustreError: 362977:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.143@tcp: cfg command failed: rc = -17 [12173.446742] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.206.143@tcp [12173.446742] [12476.903443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12476.914000] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12476.926399] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12476.945610] Lustre: Skipped 3 previous similar messages [12481.480637] Lustre: server umount lustre-MDT0001 complete [12482.035534] LustreError: 358286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12482.072596] LustreError: 358286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12493.489625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12493.587626] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12493.591447] Lustre: Found index 1 for lustre-MDT0001, updating log [12493.635645] Lustre: 358275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12493.933393] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12498.726750] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12499.439790] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12500.995392] LustreError: 371486:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12501.005580] LustreError: 371486:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12501.016755] LustreError: 358277:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12501.022440] LustreError: 358277:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12501.126945] LustreError: 371497:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12501.155765] LustreError: 371497:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.143@tcp: cfg command failed: rc = -114 [12501.163920] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12501.163920] [12821.478373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12821.496485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12821.518567] Lustre: Skipped 1 previous similar message [12821.524756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12821.529172] Lustre: Skipped 1 previous similar message [12827.439430] Lustre: server umount lustre-MDT0000 complete [12831.967480] LustreError: 361474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451068 with bad export cookie 15201457945587951733 [12831.969368] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12831.987240] LustreError: 361474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12832.233126] LustreError: 378569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12832.235658] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12832.260217] LustreError: 378569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12832.297229] Lustre: Skipped 2 previous similar messages [12832.303589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12832.312497] Lustre: Skipped 5 previous similar messages [12838.718224] Lustre: server umount lustre-MDT0001 complete [12843.208238] Lustre: server umount lustre-OST0000 complete [12853.415748] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [12864.744458] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [12906.682339] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 11:59:01 (1788451141) [12915.119896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12915.389493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12915.411961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12915.515246] Lustre: lustre-MDT0000: new disk, initializing [12915.598154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12915.619231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12921.340877] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12933.286110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12933.389935] Lustre: 381889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12933.453646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12933.460209] Lustre: Skipped 1 previous similar message [12933.552495] Lustre: lustre-MDT0001: new disk, initializing [12933.681591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12933.748242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12933.759776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12938.997255] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12948.809269] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12954.723990] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12961.669972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12961.963594] Lustre: lustre-OST0000: new disk, initializing [12961.975019] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12961.979801] Lustre: 383856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12962.077759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12968.433741] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_default_debug -1 all [12971.102083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12971.112257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12971.273660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12978.852649] Lustre: DEBUG MARKER: oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12986.848721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12986.861278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12986.877148] Lustre: Skipped 2 previous similar messages [12986.885099] Lustre: Skipped 3 previous similar messages [12988.574794] Lustre: server umount lustre-MDT0000 complete [12991.981061] LustreError: 381900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12992.005774] LustreError: 381900:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12992.587122] LustreError: 381881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451229 with bad export cookie 15201457945587953679 [12992.594751] LustreError: 382695:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12992.598502] LustreError: MGC192.168.206.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12993.017882] Lustre: server umount lustre-MDT0001 complete [13007.357182] Lustre: server umount lustre-OST0000 complete [13037.678665] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing set_hostid [13047.015431] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing load_modules_local [13079.516213] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12805 sec ========= 12:01:54 (1788451314) [13081.397714] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:01:56 (1788451316) === [13084.787911] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:02:00 (1788451320) === [13126.251517] Lustre: DEBUG MARKER: oleg643-server.virtnet: executing unload_modules_local [13129.659525] Key type lgssc unregistered [13130.024197] LNet: 390233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13130.046916] LNetError: 390233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13130.068925] LNet: Removed LNI 192.168.206.143@tcp [13130.988395] Key type .llcrypt unregistered [13130.990324] Key type ._llcrypt unregistered