[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479780556 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2528MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002314] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006902] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010156] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.021189] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025735] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029119] *** VALIDATE tmpfs *** [ 0.030497] *** VALIDATE proc *** [ 0.032246] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.035005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.042778] debug: unmapping init [mem 0xffffffffa5a59000-0xffffffffa5a60fff] [ 0.044187] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045739] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046027] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053262] rcu: Hierarchical SRCU implementation. [ 0.055294] smp: Bringing up secondary CPUs ... [ 0.056522] x86: Booting SMP configuration: [ 0.057028] .... node #0, CPUs: #1 #2 #3 [ 0.061021] smp: Brought up 1 node, 4 CPUs [ 0.063018] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.164328] node 0 deferred pages initialised in 98ms [ 0.168196] devtmpfs: initialized [ 0.169268] x86/mm: Memory block size: 128MB [ 0.171720] gcov: version magic: 0x41383552 [ 0.175387] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.176104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.177315] pinctrl core: initialized pinctrl subsystem [ 0.178236] [ 0.178756] ************************************************************* [ 0.180018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183014] ** ** [ 0.185014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.187014] ** ** [ 0.189016] ** This means that this kernel is built to expose internal ** [ 0.191015] ** IOMMU data structures, which may compromise security on ** [ 0.194017] ** your system. ** [ 0.196013] ** ** [ 0.199018] ** If you see this message and you are not debugging the ** [ 0.201014] ** kernel, report this immediately to your vendor! ** [ 0.204018] ** ** [ 0.206014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208016] ************************************************************* [ 0.210713] NET: Registered protocol family 16 [ 0.212476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.214099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.217078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.220112] cpuidle: using governor menu [ 0.222618] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.225514] PCI: Using configuration type 1 for base access [ 0.227127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.235065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.238040] cryptd: max_cpu_qlen set to 1000 [ 0.240236] ACPI: Added _OSI(Module Device) [ 0.242023] ACPI: Added _OSI(Processor Device) [ 0.244016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.246015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.251387] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.259249] ACPI: Interpreter enabled [ 0.260069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262015] ACPI: Using IOAPIC for interrupt routing [ 0.264145] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.267407] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.277969] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.280042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.286084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290298] acpiphp: Slot [2] registered [ 0.292137] acpiphp: Slot [5] registered [ 0.293106] acpiphp: Slot [6] registered [ 0.295110] acpiphp: Slot [7] registered [ 0.296100] acpiphp: Slot [8] registered [ 0.297114] acpiphp: Slot [9] registered [ 0.299127] acpiphp: Slot [10] registered [ 0.300124] acpiphp: Slot [3] registered [ 0.302085] acpiphp: Slot [4] registered [ 0.303080] acpiphp: Slot [11] registered [ 0.304103] acpiphp: Slot [12] registered [ 0.306104] acpiphp: Slot [13] registered [ 0.307102] acpiphp: Slot [14] registered [ 0.309100] acpiphp: Slot [15] registered [ 0.310121] acpiphp: Slot [16] registered [ 0.311105] acpiphp: Slot [17] registered [ 0.313114] acpiphp: Slot [18] registered [ 0.314103] acpiphp: Slot [19] registered [ 0.316100] acpiphp: Slot [20] registered [ 0.317134] acpiphp: Slot [21] registered [ 0.318124] acpiphp: Slot [22] registered [ 0.320102] acpiphp: Slot [23] registered [ 0.321122] acpiphp: Slot [24] registered [ 0.323123] acpiphp: Slot [25] registered [ 0.324140] acpiphp: Slot [26] registered [ 0.325129] acpiphp: Slot [27] registered [ 0.327113] acpiphp: Slot [28] registered [ 0.328103] acpiphp: Slot [29] registered [ 0.330133] acpiphp: Slot [30] registered [ 0.331121] acpiphp: Slot [31] registered [ 0.333060] PCI host bridge to bus 0000:00 [ 0.334016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.338024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350045] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353315] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367706] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377650] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382742] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387775] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.420234] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.463021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.474187] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.485016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.498041] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.524018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.537231] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.545026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.556090] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.581021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.593240] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.605023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.617018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.652020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.664880] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.674018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.683019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.734022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.748812] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.759017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.767020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.795027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.814534] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.817481] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.819467] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.822414] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.825224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.830180] iommu: Default domain type: Passthrough [ 0.832392] SCSI subsystem initialized [ 0.834131] ACPI: bus type USB registered [ 0.835106] usbcore: registered new interface driver usbfs [ 0.837078] usbcore: registered new interface driver hub [ 0.838088] usbcore: registered new device driver usb [ 0.840180] pps_core: LinuxPPS API ver. 1 registered [ 0.842011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.845066] PTP clock support registered [ 0.847075] EDAC MC: Ver: 3.0.0 [ 0.849119] PCI: Using ACPI for IRQ routing [ 0.850939] NetLabel: Initializing [ 0.851011] NetLabel: domain hash size = 128 [ 0.852008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.853094] NetLabel: unlabeled traffic allowed by default [ 0.855093] vgaarb: loaded [ 0.856190] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.858011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.864309] clocksource: Switched to clocksource kvm-clock [ 0.971602] VFS: Disk quotas dquot_6.6.0 [ 0.973160] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.975538] *** VALIDATE ramfs *** [ 0.976787] *** VALIDATE hugetlbfs *** [ 0.979085] pnp: PnP ACPI init [ 0.981534] pnp: PnP ACPI: found 6 devices [ 1.004633] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.007816] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.009914] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.011927] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.014715] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.017315] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.020269] NET: Registered protocol family 2 [ 1.022535] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.026743] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.029947] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.034689] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.037905] TCP: Hash tables configured (established 65536 bind 65536) [ 1.040494] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.043132] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.045634] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.048485] NET: Registered protocol family 1 [ 1.051229] RPC: Registered named UNIX socket transport module. [ 1.053571] RPC: Registered udp transport module. [ 1.054732] RPC: Registered tcp transport module. [ 1.055933] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.057709] NET: Registered protocol family 44 [ 1.059244] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.060688] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.062515] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.064249] PCI: CLS 0 bytes, default 64 [ 1.065493] Unpacking initramfs... [ 2.531487] debug: unmapping init [mem 0xffff9ceffcc54000-0xffff9ceffffbffff] [ 2.542316] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.544653] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.550346] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.075811] Initialise system trusted keyrings [ 3.077396] Key type blacklist registered [ 3.080101] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.091878] zbud: loaded [ 3.095637] *** VALIDATE nfs *** [ 3.096748] *** VALIDATE nfs4 *** [ 3.098541] pstore: using deflate compression [ 3.102595] Platform Keyring initialized [ 3.229748] NET: Registered protocol family 38 [ 3.231597] Key type asymmetric registered [ 3.233032] Asymmetric key parser 'x509' registered [ 3.234801] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.238085] io scheduler mq-deadline registered [ 3.239619] io scheduler kyber registered [ 3.241123] io scheduler bfq registered [ 3.242910] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.245886] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.248553] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.251296] ACPI: Power Button [PWRF] [ 3.257399] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.264942] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.280194] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.289205] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.310475] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.337346] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.366087] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.371635] Non-volatile memory driver v1.3 [ 3.373810] Linux agpgart interface v0.103 [ 3.410550] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.414589] vda: detected capacity change from 0 to 76779520 [ 3.430567] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.434651] vdb: detected capacity change from 0 to 1073741824 [ 3.456720] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.460410] vdc: detected capacity change from 0 to 2621440000 [ 3.480892] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.483295] vdd: detected capacity change from 0 to 2621440000 [ 3.502936] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.505747] vde: detected capacity change from 0 to 4294967296 [ 3.524212] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.527017] vdf: detected capacity change from 0 to 4294967296 [ 3.540322] libphy: Fixed MDIO Bus: probed [ 3.554457] usbcore: registered new interface driver usbserial_generic [ 3.557171] usbserial: USB Serial support registered for generic [ 3.559958] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.563806] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.565204] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.567246] mousedev: PS/2 mouse device common for all mice [ 3.571240] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.572937] rtc_cmos 00:05: RTC can wake from S4 [ 3.577771] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.582485] rtc_cmos 00:05: registered as rtc0 [ 3.584638] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.585382] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.586863] intel_pstate: CPU model not supported [ 3.592078] hid: raw HID events driver (C) Jiri Kosina [ 3.593631] usbcore: registered new interface driver usbhid [ 3.595194] usbhid: USB HID core driver [ 3.596413] drop_monitor: Initializing network drop monitor service [ 3.598122] Initializing XFRM netlink socket [ 3.599658] NET: Registered protocol family 10 [ 3.602072] Segment Routing with IPv6 [ 3.603295] NET: Registered protocol family 17 [ 3.605607] mpls_gso: MPLS GSO support [ 3.611196] RAS: Correctable Errors collector initialized. [ 3.612685] AVX version of gcm_enc/dec engaged. [ 3.613752] AES CTR mode by8 optimization enabled [ 3.690504] sched_clock: Marking stable (3690481347, 0)->(4580767893, -890286546) [ 3.695062] registered taskstats version 1 [ 3.697087] Loading compiled-in X.509 certificates [ 3.698607] zswap: loaded using pool lzo/zbud [ 3.726131] Key type big_key registered [ 3.739721] Key type encrypted registered [ 3.741419] ima: No TPM chip found, activating TPM-bypass! [ 3.743622] ima: Allocated hash algorithm: sha1 [ 3.745362] ima: No architecture policies found [ 3.747253] evm: Initialising EVM extended attributes: [ 3.749183] evm: security.selinux [ 3.750313] evm: security.ima [ 3.751475] evm: security.capability [ 3.752835] evm: HMAC attrs: 0x1 [ 3.755257] rtc_cmos 00:05: setting system clock to 2026-09-14 16:31:11 UTC (1789403471) [ 3.761861] debug: unmapping init [mem 0xffffffffa6a03000-0xffffffffa6bfffff] [ 3.765326] debug: unmapping init [mem 0xffffffffa5782000-0xffffffffa5a58fff] [ 3.775108] Write protecting the kernel read-only data: 28672k [ 3.778300] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 3.780229] debug: unmapping init [mem 0xffffffffa4714000-0xffffffffa47fffff] [ 3.816562] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.825649] systemd[1]: Detected virtualization kvm. [ 3.826935] systemd[1]: Detected architecture x86-64. [ 3.828587] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.855116] systemd[1]: No hostname configured. [ 3.857118] systemd[1]: Set hostname to . [ 3.859448] random: systemd: uninitialized urandom read (16 bytes read) [ 3.861597] systemd[1]: Initializing machine ID from random generator. [ 3.902871] random: ln: uninitialized urandom read (6 bytes read) [ 3.995234] random: systemd: uninitialized urandom read (16 bytes read) [ 3.997662] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.005048] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.010261] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.678748] device-mapper: uevent: version 1.0.3 [ 4.680951] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.458534] random: fast init done [ 5.467948] virtio_net virtio0 ens2: renamed from eth0 [ 5.503281] scsi host0: ata_piix [ 5.547340] scsi host1: ata_piix [ 5.549405] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.551976] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.945737] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.259861] random: crng init done [ 10.261270] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.648769] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.703694] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.992646] SELinux: Disabled at runtime. [ 12.054224] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.062409] systemd[1]: Detected virtualization kvm. [ 12.064476] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.519788] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.523182] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.527951] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.532245] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.535703] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.542785] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.550606] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ 12.651288] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.923252] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.234372] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.280393] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.417490] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.427191] EDAC sbridge: Ver: 1.1.2 [ 15.044502] Key type dns_resolver registered [ 15.357580] NFS: Registering the id_resolver key type [ 15.359597] Key type id_resolver registered [ 15.360701] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg107-server login: [ 45.629677] libcfs: loading out-of-tree module taints kernel. [ 45.677799] Key type ._llcrypt registered [ 45.681458] Key type .llcrypt registered [ 45.779467] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 66.309803] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 68.288551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.315382] alg: No test for adler32 (adler32-zlib) [ 69.630375] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 70.943642] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 72.815941] Key type lgssc registered [ 74.712043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 84.825888] hrtimer: interrupt took 24890955 ns [ 99.939623] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.878658] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 159.931411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 159.957145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.279051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.309211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.387638] Lustre: lustre-MDT0000: new disk, initializing [ 161.483932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.506088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 165.872558] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.353900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.451685] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.495070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.502383] Lustre: Skipped 1 previous similar message [ 179.615996] Lustre: lustre-MDT0001: new disk, initializing [ 179.724154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.764161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.776449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.446856] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.035337] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.339329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.732935] Lustre: lustre-OST0000: new disk, initializing [ 199.735578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.739629] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.794401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.349170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.357724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.474636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.119533] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.604575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.754969] Lustre: lustre-OST0001: new disk, initializing [ 221.762772] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.767961] Lustre: 9518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.825784] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 229.443113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.446627] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.613779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 230.087714] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.706559] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.480451] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.533171] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.380835] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing yml_node [ 269.985527] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 272.851758] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 276.167743] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 277.913665] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 12:35:43 EDT 2026 [ 297.651037] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 299.494754] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.913559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.922692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.924333] Lustre: Skipped 3 previous similar messages [ 316.933156] Lustre: Skipped 2 previous similar messages [ 321.391618] Lustre: server umount lustre-MDT0000 complete [ 325.311376] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789403793 with bad export cookie 3267218482228429900 [ 325.321056] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 325.326747] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 327.136748] LustreError: 6515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.140153] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.152320] LustreError: 6515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 327.157546] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 327.170447] Lustre: Skipped 1 previous similar message [ 327.180323] Lustre: Skipped 1 previous similar message [ 330.144322] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.144752] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.167699] LustreError: 6514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 330.193595] Lustre: Skipped 2 previous similar messages [ 331.841507] Lustre: server umount lustre-MDT0001 complete [ 336.181248] Lustre: server umount lustre-OST0000 complete [ 339.993200] Lustre: server umount lustre-OST0001 complete [ 347.256436] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 356.265126] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 390.188871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.509435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 390.547875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 390.731397] Lustre: lustre-MDT0000: new disk, initializing [ 390.882965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.904989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 395.149446] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 406.086177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.188990] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 406.245055] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 406.248930] Lustre: Skipped 1 previous similar message [ 406.343470] Lustre: lustre-MDT0001: new disk, initializing [ 406.431426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 406.491365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 406.509494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 410.959930] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 421.026608] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 427.149800] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 435.443226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 435.945721] Lustre: lustre-OST0000: new disk, initializing [ 435.949061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 435.956144] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 436.128735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.714202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 437.742735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 438.109047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 445.835210] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 456.075040] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 461.754159] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 462.048353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 466.961322] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 467.101229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 474.091078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.099101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.104069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.118702] Lustre: Skipped 1 previous similar message [ 475.339382] Lustre: server umount lustre-OST0000 complete [ 483.300534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 483.332269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.369860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 483.383553] Lustre: Skipped 1 previous similar message [ 485.920986] Lustre: server umount lustre-MDT0000 complete [ 489.445509] LustreError: 15077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.481260] LustreError: 15077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 491.060421] LustreError: 15876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789403958 with bad export cookie 3267218482228431426 [ 491.066733] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 491.089110] LustreError: 15876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 491.642434] Lustre: server umount lustre-MDT0001 complete [ 499.847727] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 12:39:24 (1789403964) [ 508.288908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.937564] LustreError: 19305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.018441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.025199] LustreError: 19306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.184082] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 524.257026] LustreError: 19305:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.269467] LustreError: 19305:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 525.327958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.739503] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 536.680593] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 541.909596] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 549.895865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 556.971654] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 565.271177] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 570.304899] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 579.047810] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.065816] Lustre: Skipped 1 previous similar message [ 579.077033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 579.083957] Lustre: Skipped 3 previous similar messages [ 584.119285] Lustre: server umount lustre-MDT0000 complete [ 584.165683] LustreError: 19302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.179191] LustreError: 19302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 587.962721] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404055 with bad export cookie 3267218482228432315 [ 587.971624] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.980086] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 588.462949] Lustre: server umount lustre-MDT0001 complete [ 601.625651] Lustre: server umount lustre-OST0000 complete [ 627.323406] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 640.758630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.525184] LustreError: 23687:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.604802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.621081] Lustre: Skipped 2 previous similar messages [ 646.865482] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 657.252178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.667602] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 667.270907] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 677.130225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 685.444479] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 695.042916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 695.276266] Lustre: lustre-OST0001: new disk, initializing [ 695.281332] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 695.286416] Lustre: 26009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 696.978559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 696.990507] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 697.097897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 702.521748] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 712.747972] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 716.608108] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 729.057622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 729.085511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.106235] Lustre: Skipped 2 previous similar messages [ 729.113509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 734.347234] Lustre: server umount lustre-MDT0000 complete [ 737.926239] LustreError: 26008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404205 with bad export cookie 3267218482228433162 [ 737.930952] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 737.939116] LustreError: 26008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 738.308784] Lustre: server umount lustre-MDT0001 complete [ 752.882336] Lustre: server umount lustre-OST0000 complete [ 754.608138] Lustre: 3640:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789404206/real 1789404206] req@ffff9cf0674e8e00 x1876325604512128/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789404222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 754.626190] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.637140] Lustre: Skipped 3 previous similar messages [ 757.145261] Lustre: server umount lustre-OST0001 complete [ 774.674703] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 784.257125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.534493] LustreError: 29027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.542056] LustreError: 29027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 784.635303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.643312] Lustre: Skipped 3 previous similar messages [ 788.841469] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 796.973102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 801.636947] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 804.848476] Lustre: 30174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 811.038963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.555747] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 829.728445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 829.926976] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up) [ 829.938776] Lustre: Skipped 4 previous similar messages [ 836.783340] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 845.959294] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 849.873638] Lustre: 32051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 858.593429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 858.600277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.448977] Lustre: server umount lustre-MDT0000 complete [ 866.437770] LustreError: 29007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404334 with bad export cookie 3267218482228434457 [ 866.440151] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 866.444341] LustreError: 29007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 866.887083] Lustre: server umount lustre-MDT0001 complete [ 880.743260] Lustre: server umount lustre-OST0000 complete [ 893.606233] Lustre: server umount lustre-OST0001 complete [ 902.566954] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 12:46:08 (1789404368) [ 930.622954] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 941.390744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.918140] LustreError: 34855:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 941.941765] LustreError: 34855:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 946.618793] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 955.145479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.851809] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 964.560166] Lustre: 36004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 972.724938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 981.511527] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 991.647187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 998.932702] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1007.619276] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1011.456456] Lustre: 37886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1016.802244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1016.814649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.826167] Lustre: Skipped 3 previous similar messages [ 1016.836182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.847287] Lustre: Skipped 5 previous similar messages [ 1022.260345] Lustre: server umount lustre-MDT0000 complete [ 1025.926733] LustreError: 34836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404493 with bad export cookie 3267218482228435563 [ 1025.929229] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1025.944129] LustreError: 34836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1026.386032] Lustre: server umount lustre-MDT0001 complete [ 1040.943880] Lustre: server umount lustre-OST0000 complete [ 1043.935174] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789404495/real 1789404495] req@ffff9cf070dc0380 x1876325604614912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789404511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1044.926391] Lustre: server umount lustre-OST0001 complete [ 1054.402267] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 12:48:40 (1789404520) [ 1071.343440] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1083.707753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.507122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.515944] Lustre: Skipped 7 previous similar messages [ 1090.286547] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1103.738925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.355769] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1116.673804] Lustre: 41177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1124.865664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1132.824392] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1145.420659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1153.327874] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1161.606831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1165.920555] Lustre: 43055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1166.684541] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1166.689818] Lustre: Skipped 1 previous similar message [ 1170.401804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1170.422547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.439578] Lustre: Skipped 4 previous similar messages [ 1170.451360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1170.454184] Lustre: Skipped 4 previous similar messages [ 1175.824357] Lustre: server umount lustre-MDT0000 complete [ 1179.740605] LustreError: 40007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404647 with bad export cookie 3267218482228436669 [ 1179.747639] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.755215] LustreError: 40007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1186.231949] Lustre: server umount lustre-MDT0001 complete [ 1196.166860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.218127] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9cf06b768000 x1876325604672000/t0(0) o250->MGC192.168.201.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1205.635339] LustreError: 41541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.661566] LustreError: 41541:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 33 previous similar messages [ 1210.720559] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1215.983951] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1219.156646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.429374] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1225.202972] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1225.217061] Lustre: Skipped 1 previous similar message [ 1232.535641] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1234.345046] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1241.285251] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1242.959811] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1250.272337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1255.589659] Lustre: server umount lustre-MDT0000 complete [ 1260.637076] LustreError: 43057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404728 with bad export cookie 3267218482228437796 [ 1260.658360] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1260.912851] Lustre: server umount lustre-MDT0001 complete [ 1273.560918] Lustre: server umount lustre-OST0000 complete [ 1276.896376] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789404728/real 1789404728] req@ffff9cf070dc2680 x1876325604708352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789404744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1277.254407] Lustre: server umount lustre-OST0001 complete [ 1286.186719] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 12:52:32 (1789404752) [ 1302.294487] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1315.027537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1321.577606] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1332.064268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.638795] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1341.476411] Lustre: 48815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1349.798376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1357.861977] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1367.354599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1375.437617] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1384.672775] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1391.031847] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1422.303492] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1422.649497] Lustre: server umount lustre-MDT0000 complete [ 1427.166479] LustreError: 50814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789404894 with bad export cookie 3267218482228438587 [ 1427.168584] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1427.173195] LustreError: 50814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1427.586512] Lustre: server umount lustre-MDT0001 complete [ 1442.154055] Lustre: server umount lustre-OST0000 complete [ 1445.343192] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789404897/real 1789404897] req@ffff9cf07d9d5c00 x1876325604776192/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789404913 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1445.378340] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.395361] Lustre: Skipped 14 previous similar messages [ 1446.394993] Lustre: server umount lustre-OST0001 complete [ 1457.569609] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 12:55:22 (1789404922) [ 1460.433577] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1462.714717] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 12:55:28 (1789404928) [ 1475.582587] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 1485.489748] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1510.465627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.696527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1510.779733] Lustre: lustre-MDT0000: new disk, initializing [ 1510.889646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1515.609507] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1526.840582] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1534.920579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.253881] Lustre: lustre-OST0000: new disk, initializing [ 1535.259654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1535.264446] Lustre: Skipped 1 previous similar message [ 1535.269248] Lustre: 54898:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1536.843186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1536.864859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1536.924348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1542.660931] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1553.451537] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1556.324658] Lustre: DEBUG MARKER: create test files [ 1556.702044] LustreError: 55562:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 1556.707211] LustreError: 55562:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1557.333853] LustreError: 53907:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 1557.345778] LustreError: 53907:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 2 previous similar messages [ 1557.354463] LustreError: 53907:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1557.363071] LustreError: 53907:0:(mdt_open.c:646:mdt_finish_open()) Skipped 2 previous similar messages [ 1558.668851] LustreError: 53906:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 1558.687195] LustreError: 53906:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 4 previous similar messages [ 1558.697020] LustreError: 53906:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1558.712100] LustreError: 53906:0:(mdt_open.c:646:mdt_finish_open()) Skipped 4 previous similar messages [ 1560.823309] LustreError: 55562:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 1560.838074] LustreError: 55562:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 9 previous similar messages [ 1560.847491] LustreError: 55562:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1560.855941] LustreError: 55562:0:(mdt_open.c:646:mdt_finish_open()) Skipped 9 previous similar messages [ 1564.916055] LustreError: 53905:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 1564.924762] LustreError: 53905:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 18 previous similar messages [ 1564.936353] LustreError: 53905:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1564.940648] LustreError: 53905:0:(mdt_open.c:646:mdt_finish_open()) Skipped 18 previous similar messages [ 1573.044253] LustreError: 53905:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 1573.056817] LustreError: 53905:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 28 previous similar messages [ 1573.064396] LustreError: 53905:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1573.076437] LustreError: 53905:0:(mdt_open.c:646:mdt_finish_open()) Skipped 28 previous similar messages [ 1598.437606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1598.448885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1598.469925] Lustre: Skipped 26 previous similar messages [ 1603.225699] Lustre: server umount lustre-OST0000 complete [ 1608.126732] Lustre: server umount lustre-MDT0000 complete [ 1621.989022] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1625.804726] Key type lgssc unregistered [ 1626.187394] LNet: 56525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.199881] LNetError: 56525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.237892] LNet: Removed LNI 192.168.201.107@tcp [ 1628.446229] Key type .llcrypt unregistered [ 1628.451095] Key type ._llcrypt unregistered [ 1659.753466] Key type ._llcrypt registered [ 1659.755738] Key type .llcrypt registered [ 1659.837357] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1660.946339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1661.192910] alg: No test for adler32 (adler32-zlib) [ 1662.185371] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 1662.347917] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1663.991508] Key type lgssc registered [ 1665.104392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1673.686324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1673.706607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.250566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1680.471959] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1688.880402] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1697.345459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1697.849123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.888853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1700.033621] Lustre: 59264:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1700.044695] Lustre: 59264:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1700.057525] Lustre: 59264:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1700.061625] Lustre: 59264:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1700.069919] Lustre: 59264:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1700.073425] Lustre: 59264:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1704.197175] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1713.021227] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1718.918494] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1725.479703] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1725.784698] LustreError: 58345:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 1725.792282] LustreError: 58345:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1726.432996] LustreError: 58347:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 1726.449327] LustreError: 58347:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 1726.464408] LustreError: 58347:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1726.479574] LustreError: 58347:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 1727.647175] LustreError: 58347:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 1727.671441] LustreError: 58347:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 3 previous similar messages [ 1727.678609] LustreError: 58347:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 1727.687802] LustreError: 58347:0:(mdt_open.c:646:mdt_finish_open()) Skipped 3 previous similar messages [ 1735.650657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1735.671157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1735.696195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1740.772751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1741.316638] Lustre: server umount lustre-OST0000 complete [ 1746.547550] Lustre: server umount lustre-MDT0000 complete [ 1758.648632] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1761.565752] Key type lgssc unregistered [ 1761.975127] LNet: 60870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.984634] LNetError: 60870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.046869] LNet: Removed LNI 192.168.201.107@tcp [ 1764.049699] Key type .llcrypt unregistered [ 1764.057436] Key type ._llcrypt unregistered [ 1797.932301] Key type ._llcrypt registered [ 1797.934854] Key type .llcrypt registered [ 1798.034342] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1799.793573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1799.819905] alg: No test for adler32 (adler32-zlib) [ 1800.998899] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 1801.277649] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1802.991254] Key type lgssc registered [ 1804.239041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1812.659609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1812.677615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.035308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.981752] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1826.780814] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1834.724979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1835.129410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.248061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1839.356472] Lustre: 63716:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1839.367399] Lustre: 63716:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1839.376832] Lustre: 63716:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1839.387806] Lustre: 63716:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1839.397302] Lustre: 63716:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1839.411391] Lustre: 63716:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1844.563205] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 1854.559805] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1861.175799] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1875.427818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1875.434792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.450101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.281266] Lustre: server umount lustre-OST0000 complete [ 1881.803655] Lustre: server umount lustre-MDT0000 complete [ 1893.972921] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 1897.371801] Key type lgssc unregistered [ 1897.791677] LNet: 65273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1897.796995] LNetError: 65273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.859511] LNet: Removed LNI 192.168.201.107@tcp [ 1900.139097] Key type .llcrypt unregistered [ 1900.141435] Key type ._llcrypt unregistered [ 1923.072444] Key type ._llcrypt registered [ 1923.074626] Key type .llcrypt registered [ 1923.213831] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 1941.763655] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 1943.517907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1943.557642] alg: No test for adler32 (adler32-zlib) [ 1944.620223] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 1944.827529] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 1946.528335] Key type lgssc registered [ 1947.622865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1983.679692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1983.738926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.003866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1985.090880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1985.212069] Lustre: lustre-MDT0000: new disk, initializing [ 1985.333094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.360410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1990.394629] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2002.178581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.337424] Lustre: 68426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2002.451833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2002.455533] Lustre: Skipped 1 previous similar message [ 2002.477743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2002.561572] Lustre: lustre-MDT0001: new disk, initializing [ 2002.750158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2002.809822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2002.823954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2008.164705] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2019.685344] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2026.645384] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2034.333263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2034.617357] Lustre: lustre-OST0000: new disk, initializing [ 2034.621581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2034.631824] Lustre: 70393:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2034.709820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2041.420134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2041.437450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2041.568637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2042.606886] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2053.389586] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2059.391656] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2059.711037] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2064.491491] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2064.694766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2067.424797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2067.436853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2067.437213] LustreError: Skipped 1 previous similar message [ 2067.447546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2067.452076] Lustre: Skipped 1 previous similar message [ 2067.486113] Lustre: Skipped 1 previous similar message [ 2072.549270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2072.565904] Lustre: Skipped 1 previous similar message [ 2077.675084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2077.690926] Lustre: Skipped 1 previous similar message [ 2081.247682] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2081.446842] Lustre: server umount lustre-OST0000 complete [ 2087.905916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2087.929523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.430415] Lustre: server umount lustre-MDT0000 complete [ 2096.120961] LustreError: 68418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789405563 with bad export cookie 9489799688016492260 [ 2096.123761] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.136774] LustreError: 68418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2096.396509] Lustre: server umount lustre-MDT0001 complete [ 2106.064276] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 13:06:11 (1789405571) [ 2117.722643] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2162.617054] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 2173.617032] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2213.865644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.228334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2214.302436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2214.442216] Lustre: lustre-MDT0000: new disk, initializing [ 2214.556610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.583828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2220.281196] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2232.978276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.223858] Lustre: 76415:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2233.362624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2233.367469] Lustre: Skipped 1 previous similar message [ 2233.518525] Lustre: lustre-MDT0001: new disk, initializing [ 2233.662518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2233.736916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2233.755734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2239.420691] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2250.317966] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2257.398610] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2265.348780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2265.674477] Lustre: lustre-OST0000: new disk, initializing [ 2265.679421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2265.687528] Lustre: 78383:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2265.833595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.346326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2272.361094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2272.515027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2272.952198] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2283.807921] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2288.947183] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2289.290599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2294.951173] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2295.268903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2298.339138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.352400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.363203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2298.373601] Lustre: Skipped 2 previous similar messages [ 2298.379741] Lustre: Skipped 1 previous similar message [ 2303.471052] LustreError: 78367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.498679] LustreError: 78367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2303.662572] Lustre: server umount lustre-OST0000 complete [ 2308.577030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.595607] Lustre: Skipped 1 previous similar message [ 2314.856280] Lustre: server umount lustre-MDT0000 complete [ 2318.828445] LustreError: 78659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2318.846754] LustreError: 78659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2319.092983] LustreError: 76408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789405786 with bad export cookie 9489799688016493142 [ 2319.101628] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2319.110484] LustreError: 76408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2319.830414] Lustre: server umount lustre-MDT0001 complete [ 2330.948577] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 13:09:56 (1789405796) [ 2338.420611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.839751] LustreError: 80800:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.939408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.848076] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2344.423598] LustreError: 80801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.513647] LustreError: 80800:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.759966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.161913] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2366.830988] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2373.642864] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2381.996894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2382.534597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.544548] Lustre: Skipped 1 previous similar message [ 2389.873648] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2398.086544] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2402.446300] Lustre: 83471:0:(genops.c:1773:obd_export_evict_by_uuid()) MGS: evicting 1ad95310-c8d3-4f99-aec8-c48ead5223b7 at adminstrative request [ 2403.305447] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2403.323383] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2413.549499] Lustre: MGS: Client 1ad95310-c8d3-4f99-aec8-c48ead5223b7 (at 0@lo) reconnecting [ 2413.554856] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x83b28a8981d5fdc8 to 0x83b28a8981d60068 [ 2413.565145] Lustre: MGC192.168.201.107@tcp: Connection restored to 0@lo (at 0@lo) [ 2440.803736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2441.009294] Lustre: lustre-OST0001: new disk, initializing [ 2441.017773] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2441.028240] Lustre: 83866:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2446.912362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2446.934501] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2447.109333] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2448.346241] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2459.468358] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2467.814465] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2467.830209] LustreError: Skipped 1 previous similar message [ 2467.832247] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.837818] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2467.849610] Lustre: Skipped 1 previous similar message [ 2467.866669] Lustre: Skipped 6 previous similar messages [ 2469.568161] Lustre: server umount lustre-OST0001 complete [ 2472.937949] LustreError: 83781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2472.967753] LustreError: 83781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2474.981937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.003750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.020693] Lustre: Skipped 1 previous similar message [ 2488.288047] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2488.487566] Lustre: server umount lustre-OST0000 complete [ 2496.491200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2496.507184] LustreError: Skipped 1 previous similar message [ 2496.546489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.652655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.670267] Lustre: Skipped 12 previous similar messages [ 2508.255167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2508.464525] Lustre: server umount lustre-MDT0000 complete [ 2508.769344] LustreError: 82469:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.805622] LustreError: 82469:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2514.135854] LustreError: 80783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789405981 with bad export cookie 9489799688016494696 [ 2514.138578] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2514.156462] LustreError: 80783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2515.382874] Lustre: server umount lustre-MDT0001 complete [ 2525.447501] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 13:13:10 (1789405990) [ 2527.369210] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2529.184106] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 13:13:15 (1789405995) [ 2530.929376] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2532.830223] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 13:13:18 (1789405998) [ 2534.484536] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2536.434376] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 13:13:22 (1789406002) [ 2538.137040] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2539.988825] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 13:13:25 (1789406005) [ 2541.248796] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2548.430831] loop: module loaded [ 2548.462815] loop0: detected capacity change from 0 to 4294967296 [ 2548.559188] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2550.688275] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2556.090699] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2563.197936] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2596.842765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2596.902045] LustreError: 87993:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2596.907725] LustreError: 87993:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2596.912504] LustreError: 87993:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2596.923490] LustreError: 87993:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2620.299679] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 2632.956672] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 2674.666910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.981359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2675.017585] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2675.098673] Lustre: lustre-MDT0000: new disk, initializing [ 2675.184105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.188992] Lustre: Skipped 1 previous similar message [ 2675.199911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2680.080559] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2692.489104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.687790] Lustre: 90866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2693.143438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2698.235563] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2709.065773] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2715.902574] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2724.418240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2724.822629] Lustre: 92835:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2731.062775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2731.240186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2731.437612] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2740.437746] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2744.398720] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2744.575092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2749.171794] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2749.386308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2751.970729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.975258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2751.975665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2751.980803] Lustre: Skipped 2 previous similar messages [ 2752.001212] LustreError: Skipped 1 previous similar message [ 2757.621242] Lustre: server umount lustre-OST0000 complete [ 2767.343871] LustreError: 90876:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.368747] LustreError: 90876:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2767.539859] Lustre: server umount lustre-MDT0000 complete [ 2771.161537] LustreError: 90856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789406238 with bad export cookie 9489799688016495312 [ 2771.162798] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.170883] LustreError: 90856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2771.500455] Lustre: server umount lustre-MDT0001 complete [ 2782.578490] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 13:17:28 (1789406248) [ 2790.526987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.505455] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2808.957374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.492212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2809.495734] Lustre: Skipped 3 previous similar messages [ 2815.341972] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2825.854078] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2833.111396] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2842.045881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.437767] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2860.709971] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2866.367654] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2866.616854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2871.521656] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2871.770512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2879.390544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2879.592401] Lustre: lustre-OST0001: new disk, initializing [ 2879.598113] Lustre: Skipped 2 previous similar messages [ 2879.603764] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2879.608217] Lustre: Skipped 3 previous similar messages [ 2879.613223] Lustre: 98618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2887.210911] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2888.827298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2888.836611] Lustre: Skipped 2 previous similar messages [ 2888.846710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2888.993654] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2897.911292] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2903.379936] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2903.641778] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2909.284052] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2909.549775] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2933.317333] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2937.590305] LustreError: 95242:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 2937.599927] LustreError: 95242:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 2938.126785] LustreError: 95241:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 2938.143454] LustreError: 95241:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 13 previous similar messages [ 2938.149820] LustreError: 95241:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 2938.162777] LustreError: 95241:0:(mdt_open.c:646:mdt_finish_open()) Skipped 13 previous similar messages [ 2939.150667] LustreError: 95241:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 2939.156830] LustreError: 95241:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 25 previous similar messages [ 2939.164442] LustreError: 95241:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 2939.172438] LustreError: 95241:0:(mdt_open.c:646:mdt_finish_open()) Skipped 25 previous similar messages [ 2941.161702] LustreError: 95241:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 2941.166176] LustreError: 95241:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 86 previous similar messages [ 2941.180086] LustreError: 95241:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 2941.188383] LustreError: 95241:0:(mdt_open.c:646:mdt_finish_open()) Skipped 86 previous similar messages [ 2945.178595] LustreError: 98843:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 2945.183979] LustreError: 98843:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 181 previous similar messages [ 2945.190673] LustreError: 98843:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 2945.196304] LustreError: 98843:0:(mdt_open.c:646:mdt_finish_open()) Skipped 181 previous similar messages [ 2953.180731] LustreError: 95242:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 2953.190281] LustreError: 95242:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 334 previous similar messages [ 2953.200147] LustreError: 95242:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 2953.209742] LustreError: 95242:0:(mdt_open.c:646:mdt_finish_open()) Skipped 334 previous similar messages [ 2964.831257] Lustre: Failing over lustre-MDT0000 [ 2965.212388] Lustre: server umount lustre-MDT0000 complete [ 2965.988751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.996339] LustreError: 95243:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.014907] Lustre: Skipped 3 previous similar messages [ 2966.046089] LustreError: 95243:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2981.344962] Lustre: 66646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789406433/real 1789406433] req@000000000c6f8445 x1876327569589632/t0(0) o400->MGC192.168.201.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789406449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2981.391819] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2989.772295] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2989.778397] LDISKFS-fs (dm-0): recovery complete [ 2989.790468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.594688] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83b28a8981d6cadc [ 2991.628912] Lustre: MGC192.168.201.107@tcp: Connection restored to 0@lo (at 0@lo) [ 2992.146126] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2994.069390] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2994.079718] Lustre: 95241:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2996.772954] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 2997.222674] Lustre: 95976:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2997.237556] Lustre: 95976:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 1 previous similar message [ 2997.259516] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2997.374675] LustreError: 101466:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 2997.386563] LustreError: 101466:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 355 previous similar messages [ 2997.398127] LustreError: 101466:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 2997.405253] LustreError: 101466:0:(mdt_open.c:646:mdt_finish_open()) Skipped 355 previous similar messages [ 2997.414924] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2997.741517] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2998.088399] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2998.096530] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2998.423677] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2998.427275] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 1 previous similar message [ 2998.435608] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2998.452150] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 7 previous similar messages [ 2999.415931] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2999.437237] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 3 previous similar messages [ 2999.783508] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2999.792470] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 3 previous similar messages [ 3000.443799] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3000.464623] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 5 previous similar messages [ 3001.432560] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3001.444483] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 5 previous similar messages [ 3002.071521] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3002.083868] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 6 previous similar messages [ 3004.643345] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3004.675162] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 12 previous similar messages [ 3005.650115] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3005.668015] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 3006.311123] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3006.324454] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 12 previous similar messages [ 3012.668123] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3012.695320] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 24 previous similar messages [ 3013.675117] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3013.686996] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 3014.343148] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3014.351195] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 24 previous similar messages [ 3028.720615] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3028.731468] Lustre: 101466:0:(ldlm_lib.c:2119:extend_recovery_timer()) Skipped 49 previous similar messages [ 3029.728021] LustreError: 101466:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 3029.735914] LustreError: 101466:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 49 previous similar messages [ 3029.754606] LustreError: 101466:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 3029.764145] LustreError: 101466:0:(mdt_open.c:646:mdt_finish_open()) Skipped 49 previous similar messages [ 3029.774048] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3029.788566] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 3030.423139] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3030.432505] LustreError: 101466:0:(ldlm_lib.c:2732:replay_request_or_update()) Skipped 49 previous similar messages [ 3054.255141] Lustre: 101466:0:(ldlm_lib.c:2428:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3054.257273] Lustre: 101466:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3054.260342] Lustre: 101466:0:(genops.c:1600:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client ff0e978d-a217-4596-afd9-415aad7617de@192.168.201.7@tcp [ 3054.263966] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3054.267837] Lustre: 101466:0:(ldlm_lib.c:1948:abort_req_replay_queue()) @@@ aborted: req@00000000b51e0bd4 x1876327550489728/t0(8589934774) o35->ff0e978d-a217-4596-afd9-415aad7617de@192.168.201.7@tcp:662/0 lens 392/0 e 0 to 0 dl 1789406532 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3054.284504] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3054.293429] Lustre: lustre-MDT0000: Denying connection for new client ff0e978d-a217-4596-afd9-415aad7617de (at 192.168.201.7@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3054.316864] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3054.374636] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3054.383905] Lustre: Skipped 2 previous similar messages [ 3054.390711] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3054.420801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 3054.432408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 3069.923241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3069.946238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3069.958957] Lustre: Skipped 5 previous similar messages [ 3075.661552] Lustre: server umount lustre-OST0000 complete [ 3086.856390] Lustre: server umount lustre-OST0001 complete [ 3094.515537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.524727] Lustre: Skipped 6 previous similar messages [ 3097.077301] Lustre: server umount lustre-MDT0000 complete [ 3099.618934] LustreError: 95246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.638452] LustreError: 95246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 35 previous similar messages [ 3100.506183] LustreError: 95228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789406568 with bad export cookie 9489799688016546524 [ 3100.518336] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3100.536296] LustreError: 95228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3101.150422] Lustre: server umount lustre-MDT0001 complete [ 3111.467600] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 13:22:57 (1789406577) [ 3119.357594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3119.368187] Lustre: *** cfs_fail_loc=197, val=0*** [ 3134.945227] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3140.063498] LustreError: 103447:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 3145.202356] Lustre: *** cfs_fail_loc=197, val=0*** [ 3145.205880] Lustre: Skipped 65 previous similar messages [ 3165.792036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.794964] Lustre: Skipped 3 previous similar messages [ 3172.221445] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3180.787849] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3183.200771] Lustre: server umount lustre-OST0000 complete [ 3192.548854] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 13:24:18 (1789406658) [ 3214.700287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.079217] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3229.648726] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3234.183220] LustreError: 105099:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3234.193243] LustreError: 105099:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3234.208943] LustreError: 105099:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 3234.591722] Lustre: server umount lustre-MDT0000 complete [ 3272.839584] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 3284.828779] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3328.941281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.311883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3329.381201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3329.570724] Lustre: lustre-MDT0000: new disk, initializing [ 3329.751395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3336.372654] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3350.186736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.372624] Lustre: 109274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3350.925205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3358.834459] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3370.611944] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3376.665838] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3385.023410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3385.259880] Lustre: 111245:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3392.925212] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3394.335249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3403.582712] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3409.560302] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3410.006479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3417.719984] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3418.014721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3425.254468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3425.261865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.272826] LustreError: Skipped 4 previous similar messages [ 3425.276773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3425.276783] Lustre: Skipped 9 previous similar messages [ 3425.313573] Lustre: Skipped 2 previous similar messages [ 3426.663493] Lustre: server umount lustre-OST0000 complete [ 3436.934555] Lustre: server umount lustre-MDT0000 complete [ 3440.609760] LustreError: 109282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.629452] LustreError: 109282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3442.413691] LustreError: 109269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789406910 with bad export cookie 9489799688016557157 [ 3442.422092] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3442.437584] LustreError: 109269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3442.888990] Lustre: server umount lustre-MDT0001 complete [ 3456.482229] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 13:28:41 (1789406921) [ 3470.755240] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 3474.766825] Key type lgssc unregistered [ 3475.146629] LNet: 113755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3475.165528] LNetError: 113755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3476.198638] LNet: Removed LNI 192.168.201.107@tcp [ 3477.425310] Key type .llcrypt unregistered [ 3477.432706] Key type ._llcrypt unregistered [ 3498.169499] Key type ._llcrypt registered [ 3498.172583] Key type .llcrypt registered [ 3498.323393] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 3517.586916] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3519.110071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3519.162794] alg: No test for adler32 (adler32-zlib) [ 3520.155613] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 3520.429628] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 3522.111170] Key type lgssc registered [ 3523.296987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3567.986479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3568.045241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.463773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3569.530772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3569.653539] Lustre: lustre-MDT0000: new disk, initializing [ 3569.725780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.750681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3575.888867] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3588.601349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3588.991667] Lustre: lustre-OST0000: new disk, initializing [ 3588.996578] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3589.004969] Lustre: Skipped 1 previous similar message [ 3589.016588] Lustre: 117923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3589.178311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3594.952682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3594.966333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3595.028723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3597.019263] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3610.221870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3610.447302] Lustre: lustre-OST0001: new disk, initializing [ 3610.451852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3610.457527] Lustre: 118789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3610.529697] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3617.550988] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3619.042886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3619.065500] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3619.113119] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3623.527927] LustreError: 118134:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 3623.536220] LustreError: 118134:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 3629.541137] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.547377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3629.553242] Lustre: Skipped 1 previous similar message [ 3629.561208] Lustre: Skipped 1 previous similar message [ 3632.846462] Lustre: server umount lustre-MDT0000 complete [ 3639.337499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3644.607407] LustreError: 117112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407112 with bad export cookie 12863076828416315362 [ 3644.617454] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3644.625374] LustreError: 117112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3644.792906] Lustre: server umount lustre-OST0000 complete [ 3648.999847] Lustre: server umount lustre-OST0001 complete [ 3658.673263] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 13:32:03 (1789407123) [ 3685.863574] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 13:32:31 (1789407151) [ 3698.198063] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3722.756704] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 13:33:08 (1789407188) [ 3772.738691] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 3787.614831] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 3824.179423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.532160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3824.566206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3824.681377] Lustre: lustre-MDT0000: new disk, initializing [ 3824.788203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.823214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3830.263220] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3841.485041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.584421] Lustre: 126559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3841.626885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3841.630236] Lustre: Skipped 1 previous similar message [ 3841.694280] Lustre: lustre-MDT0001: new disk, initializing [ 3841.771864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3841.808172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3841.820709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3847.351873] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3860.365389] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3870.777818] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3879.024037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3879.469830] Lustre: lustre-OST0000: new disk, initializing [ 3879.474341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3879.483221] Lustre: 128531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3879.591095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3886.704370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3886.713115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3886.849848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3887.256959] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3898.063241] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3903.804415] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3904.052948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3909.742454] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3909.972471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3912.679194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3912.683744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3912.685922] LustreError: Skipped 1 previous similar message [ 3912.686609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3912.724240] Lustre: Skipped 1 previous similar message [ 3917.794903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3917.804110] Lustre: Skipped 2 previous similar messages [ 3918.310202] Lustre: server umount lustre-OST0000 complete [ 3922.915271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.949377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3922.970093] Lustre: Skipped 1 previous similar message [ 3928.039521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3937.248840] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3937.559256] Lustre: server umount lustre-MDT0000 complete [ 3941.500179] LustreError: 128530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407409 with bad export cookie 12863076828416316587 [ 3941.509514] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3942.025202] Lustre: server umount lustre-MDT0001 complete [ 3952.833870] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 13:36:58 (1789407418) [ 3959.837564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.543276] LustreError: 130946:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3960.564179] LustreError: 130946:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3960.637325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.921977] LustreError: 130947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.695956] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3971.041498] LustreError: 130946:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.963885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3976.161939] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3976.178964] Lustre: Skipped 3 previous similar messages [ 3980.929321] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 3989.806868] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3997.224946] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4006.319203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4006.971244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4006.991613] Lustre: Skipped 1 previous similar message [ 4016.826717] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4027.088285] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4036.477623] LustreError: 130941:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [ 4036.485818] LustreError: 130941:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 4036.613854] LustreError: 130942:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 4036.618073] LustreError: 130942:0:(mdt_open.c:646:mdt_finish_open()) Skipped 2 previous similar messages [ 4038.665972] Lustre: *** cfs_fail_loc=159, val=0*** [ 4038.674238] Lustre: Skipped 1 previous similar message [ 4055.013061] Lustre: lustre-MDT0000: Client f0408aee-5ea2-43c9-8d69-59b451bd3c9a (at 192.168.201.7@tcp) reconnecting [ 4057.322950] Lustre: *** cfs_fail_loc=159, val=0*** [ 4057.324639] Lustre: Skipped 4 previous similar messages [ 4072.920341] Lustre: lustre-MDT0000: Client f0408aee-5ea2-43c9-8d69-59b451bd3c9a (at 192.168.201.7@tcp) reconnecting [ 4092.385100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4092.392879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4092.402470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4092.413363] Lustre: Skipped 2 previous similar messages [ 4097.281866] Lustre: server umount lustre-OST0000 complete [ 4102.628050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.645253] Lustre: Skipped 1 previous similar message [ 4109.180454] Lustre: server umount lustre-MDT0000 complete [ 4112.886195] LustreError: 130947:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.933226] LustreError: 130947:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4114.036868] LustreError: 130928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407581 with bad export cookie 12863076828416317469 [ 4114.048316] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4114.052716] LustreError: 130928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4114.542630] Lustre: server umount lustre-MDT0001 complete [ 4130.342729] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 4134.624907] Key type lgssc unregistered [ 4134.890465] LNet: 134737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4134.902753] LNetError: 134737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4134.919709] LNet: Removed LNI 192.168.201.107@tcp [ 4136.063717] Key type .llcrypt unregistered [ 4136.072299] Key type ._llcrypt unregistered [ 4167.761498] Key type ._llcrypt registered [ 4167.763891] Key type .llcrypt registered [ 4167.926454] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4169.546052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4169.626577] alg: No test for adler32 (adler32-zlib) [ 4170.692310] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 4171.021815] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 4172.711255] Key type lgssc registered [ 4173.528741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4183.466524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4183.492094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.894366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4190.650992] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4194.277156] LustreError: 136150:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.317096] LustreError: 136150:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4199.399557] LustreError: 136151:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4199.935969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4200.555284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4206.419822] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4215.389844] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4222.549611] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4229.847893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4230.150388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4236.419454] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4243.928930] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4254.738448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4258.877405] LustreError: 136147:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [ 4259.064444] LustreError: 136145:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 4265.335288] Lustre: *** cfs_fail_loc=159, val=0*** [ 4280.832712] Lustre: lustre-MDT0000: Client c6dc6423-c00e-4a4c-992b-434db4d8d46f (at 192.168.201.7@tcp) reconnecting [ 4281.536522] LustreError: 136146:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 4281.540631] LustreError: 136146:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 3 previous similar messages [ 4281.544894] LustreError: 136146:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 4281.553607] LustreError: 136146:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 4283.528812] Lustre: *** cfs_fail_loc=159, val=0*** [ 4299.720756] Lustre: lustre-MDT0001: Client c6dc6423-c00e-4a4c-992b-434db4d8d46f (at 192.168.201.7@tcp) reconnecting [ 4301.436253] Lustre: *** cfs_fail_loc=159, val=0*** [ 4301.444013] Lustre: Skipped 3 previous similar messages [ 4316.617562] Lustre: lustre-MDT0001: Client c6dc6423-c00e-4a4c-992b-434db4d8d46f (at 192.168.201.7@tcp) reconnecting [ 4316.824312] LustreError: 138861:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [ 4316.827943] LustreError: 138861:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 5 previous similar messages [ 4321.455188] Lustre: 139210:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4322.140509] LustreError: 137320:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [ 4322.143743] LustreError: 137320:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 4322.255795] LustreError: 137320:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 4322.261031] LustreError: 137320:0:(mdt_open.c:646:mdt_finish_open()) Skipped 5 previous similar messages [ 4324.366626] Lustre: *** cfs_fail_loc=159, val=0*** [ 4324.377376] Lustre: Skipped 4 previous similar messages [ 4339.663640] Lustre: lustre-MDT0000: Client a3b9714b-7378-411c-bd21-86ad6f3d6382 (at 192.168.201.7@tcp) reconnecting [ 4342.048932] Lustre: *** cfs_fail_loc=159, val=0*** [ 4342.060974] Lustre: Skipped 14 previous similar messages [ 4357.550477] Lustre: lustre-MDT0000: Client a3b9714b-7378-411c-bd21-86ad6f3d6382 (at 192.168.201.7@tcp) reconnecting [ 4359.080968] Lustre: 139501:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4382.688735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4382.702851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.705090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4382.734143] Lustre: Skipped 1 previous similar message [ 4385.476494] Lustre: server umount lustre-OST0000 complete [ 4389.858097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4389.868781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.887072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.891368] Lustre: Skipped 1 previous similar message [ 4392.928117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4392.928735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4395.921934] Lustre: server umount lustre-MDT0000 complete [ 4398.058259] LustreError: 138871:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.289295] LustreError: 139333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789407868 with bad export cookie 107948863004936780 [ 4400.298553] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4400.301633] LustreError: 139333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4400.758653] Lustre: server umount lustre-MDT0001 complete [ 4420.755149] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 4423.951098] Key type lgssc unregistered [ 4424.507882] LNet: 140604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4424.519453] LNetError: 140604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4424.535535] LNet: Removed LNI 192.168.201.107@tcp [ 4425.513218] Key type .llcrypt unregistered [ 4425.514637] Key type ._llcrypt unregistered [ 4455.244702] Key type ._llcrypt registered [ 4455.247588] Key type .llcrypt registered [ 4455.362677] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4456.889236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4456.964898] alg: No test for adler32 (adler32-zlib) [ 4458.054148] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 4458.436919] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 4460.095239] Key type lgssc registered [ 4461.134742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4470.484355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4470.542572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.138240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4478.425528] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4481.523530] LustreError: 142017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4481.560076] LustreError: 142017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4486.633655] LustreError: 142036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.270111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.779562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4496.865040] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4505.563498] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4512.317936] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4520.161924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4520.546312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4527.634719] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4535.934023] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4536.212716] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4542.096336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4548.018681] Lustre: 144610:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4551.907196] Lustre: 144660:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4553.665293] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4554.567498] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4559.848531] Lustre: 144806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4560.807491] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4561.663735] Lustre: 144904:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4568.057126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.057598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4568.066643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4568.092817] Lustre: Skipped 1 previous similar message [ 4573.153162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4573.168760] Lustre: Skipped 2 previous similar messages [ 4574.046340] Lustre: server umount lustre-OST0000 complete [ 4582.891715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.898196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.923832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4586.077942] Lustre: server umount lustre-MDT0000 complete [ 4588.516118] LustreError: 144520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.539276] LustreError: 144520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4590.488484] LustreError: 141996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789408058 with bad export cookie 8669020518482008988 [ 4590.489709] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4590.497695] LustreError: 141996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4590.818216] Lustre: server umount lustre-MDT0001 complete [ 4603.160594] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 4605.981401] Key type lgssc unregistered [ 4606.362850] LNet: 145964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4606.368887] LNetError: 145964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4607.398035] LNet: Removed LNI 192.168.201.107@tcp [ 4608.621966] Key type .llcrypt unregistered [ 4608.624107] Key type ._llcrypt unregistered [ 4638.966455] Key type ._llcrypt registered [ 4638.969877] Key type .llcrypt registered [ 4639.092454] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4640.144607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4640.179127] alg: No test for adler32 (adler32-zlib) [ 4641.274601] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 4641.521438] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 4643.223312] Key type lgssc registered [ 4644.176709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4652.013963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4652.067439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.670820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4658.587752] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4663.269482] LustreError: 147377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4663.284033] LustreError: 147377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4668.386258] LustreError: 147378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.476312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.972550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4673.751609] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4682.518152] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4689.865604] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4698.215371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4698.593634] LustreError: 149202:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.821741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4704.276909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4706.363631] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4713.588128] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4717.706815] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4722.449209] LustreError: 149875:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [ 4722.822769] LustreError: 147373:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 4723.002182] LustreError: 149875:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 4723.005638] LustreError: 149875:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 2 previous similar messages [ 4723.327392] LustreError: 147373:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 4723.333487] LustreError: 147373:0:(mdt_open.c:646:mdt_finish_open()) Skipped 3 previous similar messages [ 4724.528483] Lustre: *** cfs_fail_loc=159, val=0*** [ 4741.164562] Lustre: lustre-MDT0000: Client 909eaec2-3df0-4027-844b-cb715b6166a5 (at 192.168.201.7@tcp) reconnecting [ 4749.793035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4749.805926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4749.823707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4749.844302] Lustre: Skipped 1 previous similar message [ 4754.871960] Lustre: server umount lustre-OST0000 complete [ 4760.038073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4760.052182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4760.064906] Lustre: Skipped 1 previous similar message [ 4760.089155] Lustre: Skipped 1 previous similar message [ 4765.153761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4765.161321] Lustre: Skipped 1 previous similar message [ 4765.520037] Lustre: server umount lustre-MDT0000 complete [ 4769.676250] LustreError: 147359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789408237 with bad export cookie 3174977043099459182 [ 4769.689193] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4769.697216] LustreError: 147359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4770.221475] Lustre: server umount lustre-MDT0001 complete [ 4784.626699] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 4788.066943] Key type lgssc unregistered [ 4788.531275] LNet: 151042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4788.542858] LNetError: 151042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4788.582920] LNet: Removed LNI 192.168.201.107@tcp [ 4789.811201] Key type .llcrypt unregistered [ 4789.812946] Key type ._llcrypt unregistered [ 4824.668864] Key type ._llcrypt registered [ 4824.677972] Key type .llcrypt registered [ 4824.833094] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 4826.425443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4826.489136] alg: No test for adler32 (adler32-zlib) [ 4827.524439] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 4827.730122] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 4829.367221] Key type lgssc registered [ 4830.544344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4838.617363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4838.636352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.071912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.803372] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4849.632473] LustreError: 152455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4849.656705] LustreError: 152455:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4853.447196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4853.943250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4858.751760] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4867.498901] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4873.775603] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4881.502549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4881.987686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4885.115442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4888.431311] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 4898.459367] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4900.619482] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4905.169370] LustreError: 152451:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 4905.182183] LustreError: 152451:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 4925.061695] Lustre: lustre-OST0000: evicting 7cbcf48b-0dc8-4d3e-899a-f89878d5ed55 (at 192.168.201.7@tcp) by administrative request [ 4925.075432] Lustre: Skipped 1 previous similar message [ 4955.105188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4955.107945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4955.167300] Lustre: Skipped 1 previous similar message [ 4960.235198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4960.243145] Lustre: Skipped 2 previous similar messages [ 4961.561201] Lustre: server umount lustre-OST0000 complete [ 4966.369879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4966.384660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4966.408438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4970.471105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4970.478895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4975.587233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4975.600845] Lustre: Skipped 2 previous similar messages [ 4980.191160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4980.330388] Lustre: server umount lustre-MDT0000 complete [ 4980.705407] LustreError: 153468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4980.722220] LustreError: 153468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4984.653698] LustreError: 152436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789408452 with bad export cookie 9696976044375189466 [ 4984.669204] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4984.677245] LustreError: 152436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4985.122488] Lustre: server umount lustre-MDT0001 complete [ 5001.141281] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5004.709252] Key type lgssc unregistered [ 5005.018948] LNet: 156534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5005.030689] LNetError: 156534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5006.053983] LNet: Removed LNI 192.168.201.107@tcp [ 5007.222719] Key type .llcrypt unregistered [ 5007.224916] Key type ._llcrypt unregistered [ 5098.308488] Key type ._llcrypt registered [ 5098.311268] Key type .llcrypt registered [ 5098.419231] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5099.697164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5099.747779] alg: No test for adler32 (adler32-zlib) [ 5100.817329] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 5101.012878] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5102.743281] Key type lgssc registered [ 5103.926833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5113.285804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5113.329835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.904540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.267720] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5125.092443] LustreError: 159021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5125.102654] LustreError: 159021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5129.667851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.972597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5135.992744] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5146.223758] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5153.767058] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5162.716237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5163.030505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5173.015559] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5174.337177] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5182.453988] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5184.560203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5192.396535] LustreError: 159969:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 5192.410619] LustreError: 159969:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 5199.840420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5199.843789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.848208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5199.879277] Lustre: Skipped 1 previous similar message [ 5199.902326] Lustre: Skipped 1 previous similar message [ 5201.342691] Lustre: server umount lustre-OST0000 complete [ 5207.012651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5207.026761] LustreError: Skipped 1 previous similar message [ 5207.028682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5207.069597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5210.085213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5210.096463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5215.216495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5215.227643] Lustre: Skipped 2 previous similar messages [ 5219.297760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5219.304855] Lustre: Skipped 1 previous similar message [ 5220.319193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5220.672444] Lustre: server umount lustre-MDT0000 complete [ 5224.416986] LustreError: 159021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.453594] LustreError: 159021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5225.829097] LustreError: 159002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789408693 with bad export cookie 9709374091462454423 [ 5225.837469] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5225.849506] LustreError: 159002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5226.224653] Lustre: server umount lustre-MDT0001 complete [ 5240.458465] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5244.506036] Key type lgssc unregistered [ 5244.840623] LNet: 162588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5244.855814] LNetError: 162588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5244.876952] LNet: Removed LNI 192.168.201.107@tcp [ 5245.788868] Key type .llcrypt unregistered [ 5245.794087] Key type ._llcrypt unregistered [ 5275.366293] Key type ._llcrypt registered [ 5275.373851] Key type .llcrypt registered [ 5275.543289] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5277.734639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5277.938101] alg: No test for adler32 (adler32-zlib) [ 5279.907123] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 5280.207861] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5282.050260] Key type lgssc registered [ 5284.094779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5304.543652] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 13:59:30 (1789408770) [ 5327.018429] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 5341.430927] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5383.347439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5383.389413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.761286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5384.815217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5384.872985] Lustre: lustre-MDT0000: new disk, initializing [ 5384.965405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5384.981653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5389.967036] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5402.959805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.157883] Lustre: 166872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5403.301481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5403.318401] Lustre: Skipped 1 previous similar message [ 5403.482305] Lustre: lustre-MDT0001: new disk, initializing [ 5403.626894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5403.670173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5403.679824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5409.567318] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5422.418775] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5429.742578] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5438.152717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5438.566664] Lustre: lustre-OST0000: new disk, initializing [ 5438.569989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5438.577559] Lustre: 168854:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5438.664433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5445.394196] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5448.271416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5448.280828] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5448.362504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5455.004752] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5456.882480] LustreError: 167882:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 5456.889392] LustreError: 167882:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 5469.163135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.170851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5469.172718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5469.178339] Lustre: Skipped 1 previous similar message [ 5469.193906] Lustre: Skipped 1 previous similar message [ 5471.587355] Lustre: server umount lustre-OST0000 complete [ 5479.394458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5479.406780] Lustre: Skipped 1 previous similar message [ 5479.408700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5479.425188] Lustre: Skipped 1 previous similar message [ 5482.435729] Lustre: server umount lustre-MDT0000 complete [ 5485.716163] LustreError: 166867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789408953 with bad export cookie 873109670543595034 [ 5485.717220] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5485.723219] LustreError: 166867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5485.985957] Lustre: server umount lustre-MDT0001 complete [ 5496.866249] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 5499.260624] Key type lgssc unregistered [ 5499.557597] LNet: 170760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5499.566900] LNetError: 170760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5499.578174] LNet: Removed LNI 192.168.201.107@tcp [ 5500.498178] Key type .llcrypt unregistered [ 5500.509633] Key type ._llcrypt unregistered [ 5529.152820] Key type ._llcrypt registered [ 5529.158337] Key type .llcrypt registered [ 5529.289225] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 5530.456383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5530.495187] alg: No test for adler32 (adler32-zlib) [ 5531.665134] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 5532.081294] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 5533.831286] Key type lgssc registered [ 5534.944345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5543.383265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5543.420246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5544.874378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5549.786293] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5555.172425] LustreError: 172173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5555.183281] LustreError: 172173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5558.367165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.718153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5563.087886] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5571.188066] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5577.644153] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5585.159528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5585.683536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5589.760085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5592.624759] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 5602.505736] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5610.278276] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5610.303573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.318049] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5610.323282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.343207] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5611.543261] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5611.556557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5611.564022] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5611.569223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.576778] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5612.784536] LustreError: 172170:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5612.794622] LustreError: 172170:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5612.862397] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5612.892698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.914596] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5612.922290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.935962] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5613.912888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.976774] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5613.988410] LustreError: 173415:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5614.884307] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5614.893136] Lustre: Skipped 1 previous similar message [ 5614.913506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.918896] Lustre: Skipped 1 previous similar message [ 5614.921512] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5614.923928] Lustre: Skipped 1 previous similar message [ 5614.931787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.940455] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5614.943411] Lustre: Skipped 1 previous similar message [ 5615.903643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.943848] LustreError: 172170:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5615.956664] LustreError: 172170:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 45 previous similar messages [ 5615.975616] LustreError: 172170:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5615.978617] LustreError: 172170:0:(mdt_open.c:646:mdt_finish_open()) Skipped 45 previous similar messages [ 5617.035655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.925446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.053953] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5618.071864] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 41 previous similar messages [ 5618.093863] LustreError: 173415:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5618.102225] LustreError: 173415:0:(mdt_open.c:646:mdt_finish_open()) Skipped 41 previous similar messages [ 5619.035907] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5619.049566] Lustre: Skipped 3 previous similar messages [ 5619.075917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.089840] Lustre: Skipped 3 previous similar messages [ 5619.098545] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5619.104602] Lustre: Skipped 3 previous similar messages [ 5619.108621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.124358] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5619.133304] Lustre: Skipped 3 previous similar messages [ 5620.367219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.291905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.068691] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5622.075278] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 61 previous similar messages [ 5622.106855] LustreError: 172169:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5622.116328] LustreError: 172169:0:(mdt_open.c:646:mdt_finish_open()) Skipped 62 previous similar messages [ 5622.674969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.883189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.254630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.391270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.500754] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5627.511533] Lustre: Skipped 6 previous similar messages [ 5627.530475] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5627.546695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.563605] Lustre: Skipped 6 previous similar messages [ 5627.569753] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5627.581128] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5627.590682] Lustre: Skipped 6 previous similar messages [ 5627.600654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.615872] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5627.618920] Lustre: Skipped 6 previous similar messages [ 5628.665579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.590765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.672605] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5630.680535] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 145 previous similar messages [ 5630.692190] LustreError: 173415:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5630.697215] LustreError: 173415:0:(mdt_open.c:646:mdt_finish_open()) Skipped 144 previous similar messages [ 5630.792453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.039209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.659718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.594609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.465661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.938728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.020327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.307401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.750762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.903181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.787902] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5643.797319] Lustre: Skipped 12 previous similar messages [ 5643.815488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5643.824367] Lustre: Skipped 12 previous similar messages [ 5643.837734] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5643.847415] Lustre: Skipped 12 previous similar messages [ 5643.853716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.863905] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5643.869901] Lustre: Skipped 12 previous similar messages [ 5645.348725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.689853] LustreError: 172170:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5646.694172] LustreError: 172170:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 248 previous similar messages [ 5646.699283] LustreError: 172170:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5646.704194] LustreError: 172170:0:(mdt_open.c:646:mdt_finish_open()) Skipped 248 previous similar messages [ 5646.786807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.087872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.301868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.757142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.867543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.811821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.680820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.659595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.850741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.190717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.407146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.398182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.294895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.183242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.086425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.148900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.458820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.426576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.349985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.167754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.151868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.192435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.930273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.841367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.688909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.630273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.407871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.386445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.445984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.347194] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5676.354225] Lustre: Skipped 30 previous similar messages [ 5676.369620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.382223] Lustre: Skipped 30 previous similar messages [ 5676.385697] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5676.395226] Lustre: Skipped 30 previous similar messages [ 5676.399726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.408868] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5676.414567] Lustre: Skipped 30 previous similar messages [ 5677.576809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.656484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.631712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.755556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.849625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.702343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.627943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.782689] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5683.786777] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 579 previous similar messages [ 5683.802785] LustreError: 173415:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5683.808384] LustreError: 173415:0:(mdt_open.c:646:mdt_finish_open()) Skipped 579 previous similar messages [ 5684.483073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.406690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.203945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.247241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.166344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.052182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.963514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.707140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.591951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.337967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.286826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.133856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.073117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.910860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.727597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.559373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.489352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.491923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.491335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.581713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.401436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.549833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.701956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.578840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.575759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.598355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.666099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.768784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.976305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.755417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.918682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.788058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.965982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.569634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.500519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.838271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.061498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.389161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.662062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.061252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.430971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.624878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.087990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.261388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.195443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.451535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.374463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.992496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.492885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.447284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.652637] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5740.657297] Lustre: Skipped 57 previous similar messages [ 5740.664351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5740.670183] Lustre: Skipped 57 previous similar messages [ 5740.676611] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5740.681930] Lustre: Skipped 57 previous similar messages [ 5740.686323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.702250] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5740.721505] Lustre: Skipped 57 previous similar messages [ 5742.252373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.948426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.778270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.156126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.173465] LustreError: 172168:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5748.190172] LustreError: 172168:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1061 previous similar messages [ 5748.204220] LustreError: 172168:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5748.207552] LustreError: 172168:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1061 previous similar messages [ 5748.248885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.960890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.515782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.055795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.857411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.344870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.996572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.963427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.068535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.058708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.826931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.791446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.180161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.344445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.961988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.244618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.611800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.188155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.805777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.727477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.068637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.205336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.777663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.266721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.526760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.483944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.740919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.668632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.656285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.632622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.812944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.110669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.413881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.511829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.836383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.926682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.143965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.728278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.458349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.296914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.200587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.370982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.509534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.527409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.717922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.466621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.608205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.727131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.978990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.904919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.349305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.964099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.226807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.655955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.270492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.396380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.708563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.197727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.846052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.548651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.527648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.561783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.425542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.053818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.654088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.494323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.263669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.528852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.459731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.470151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.157512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.523918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.730393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.865507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.108068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.843879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.932306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.215708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.460975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.576219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5843.886298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.131678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.112784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.273069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.401086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.296268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5850.278304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.364254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.603637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.552960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.663640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.747404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.589528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.805552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.038362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.761994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.662776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.507563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.957931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.968433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.113259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.267566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.432802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.633481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.794304] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5869.806826] Lustre: Skipped 108 previous similar messages [ 5869.838887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.862631] Lustre: Skipped 108 previous similar messages [ 5869.878383] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5869.880743] Lustre: Skipped 108 previous similar messages [ 5869.891627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.904486] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5869.912406] Lustre: Skipped 108 previous similar messages [ 5870.990624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.032389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.970273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.121564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.979235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.128733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.146209] LustreError: 172170:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 5877.169061] LustreError: 172170:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1936 previous similar messages [ 5877.189648] LustreError: 172170:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 5877.199473] LustreError: 172170:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1936 previous similar messages [ 5877.467992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.666838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.824786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.663630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.599697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.570219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.810450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.723717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.912576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.239586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.389834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.435943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.610070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.551486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.571769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.623855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.594577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.563501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.473542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.418250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.266596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.236035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.228413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.199190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.245331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.207134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.388678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.746959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.994857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.130146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.234257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.287719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.257462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.345461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.271738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.293675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.461605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.979370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.230349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.655835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.013165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.242740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.715406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.473351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.532184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.568299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.710740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.083437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.959791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.123138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.394844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.910743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.103307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.272660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.295798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.483291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.683405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.111218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.061275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.230878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5944.541464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.437218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.735412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.947081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.223211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.421236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.431664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.678774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.815824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.843993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.163333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.688989] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5957.717820] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5957.733340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.940683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.055550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.361722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.505333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.654594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.685024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.800237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.712487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.660377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.569927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.628688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.733681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.704660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.593885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.396554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.496257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.380907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.332506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.232830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.075908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.786803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.667737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.416515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.259744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.080517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.865321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.071701] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5984.081766] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5984.084390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.895701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.872594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.722264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.736621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.815367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.106342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.310489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.367916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.377930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.521151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.498853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.648177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.790523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.762992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.997962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.997636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.076020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.200898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.308437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.260389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.156046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.048149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.147740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.222738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6010.188099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.254350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.208196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.078281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.083648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.160367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.421807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.426462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.568928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.594631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.723181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.813899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.932710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.775496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.959204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.831775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.765629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.049433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.136719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.263703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.362916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.202171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.423106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.465165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.641624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.794667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.089242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.552621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.199775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.193376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.659116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.307521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.933530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.644763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.978955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.633139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.062424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.516665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.775555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.170919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.101291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.557934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.008083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.188255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.560563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.481610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.344563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.549561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.594268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.680196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.597896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.932781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.370467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.317318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.103485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.148882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.436285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.238995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.242499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.514081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.595773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.775493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.750572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.612996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.844880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.170065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.202414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.248714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.053480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.318151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.601063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.846496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.733216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.807212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.175352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.373683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.525087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.438544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.455564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.380131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.719264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.813542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.220197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.481550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.264177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.253492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.354545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.544685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.940752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.511074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.052892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.543416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.534072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.740422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.060526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.210494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.814893] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6125.821901] Lustre: Skipped 225 previous similar messages [ 6125.866525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6125.882371] Lustre: Skipped 225 previous similar messages [ 6125.887106] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6125.902936] Lustre: Skipped 225 previous similar messages [ 6125.911637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.921809] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6125.933132] Lustre: Skipped 225 previous similar messages [ 6127.379144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.484899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.872299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.269848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.626563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.636323] LustreError: 172169:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 6133.651054] LustreError: 172169:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 3587 previous similar messages [ 6133.665296] LustreError: 172169:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 6133.681866] LustreError: 172169:0:(mdt_open.c:646:mdt_finish_open()) Skipped 3587 previous similar messages [ 6133.878459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.199550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.766676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.499956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.663400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6141.270719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.641844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.002590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.286782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.671938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.563150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.673512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.780931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.161734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.285178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.386298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.584641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.672392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.678359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.864928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.072630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.368096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.772405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.096984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.694809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.112930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.508746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.652408] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6168.662549] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6168.668869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.017749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.905633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.213686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.535135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.697962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.851648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.283838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.340124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.949368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.528767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.910624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.154303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.044343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.271435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.532481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.428732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.918347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.393379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.735646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.308298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.244166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.209465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.058328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.412072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.808977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.892169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.225143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.476301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.611473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.035497] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6206.051946] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6206.056614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.202751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.269757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.627781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.113113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.355133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.827139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.017681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.117802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.278515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.157568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.373218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.477871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.463386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.632796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.643550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.751335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.726679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.944650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.786532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.861252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.698834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.004888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.198650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.314870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.590100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.705809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.956520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.823083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.859178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.734822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.771819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.108457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.257156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.372579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.643126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.007444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.506680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.209800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.547428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.686939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.445681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.050829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.373336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.436819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.951131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.277752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.745080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.251709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.601766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.864710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.910876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.216272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.255923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.370912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.747331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.833592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.999069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.192896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.684921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.563366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.362854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.946424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.212812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.896613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.751829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.650552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.001146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.765345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.454619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.140719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.020627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.827496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.831222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.746237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.011585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.683979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.544123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.326261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.451654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.698819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.944889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.357384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.522123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.868683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.167812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.443844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.755689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.173727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.306586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.387331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.590653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.842848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.902122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.725216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.770408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.626394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.549633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.753629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.848944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.764083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.964964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.148385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.100433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.467680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.083422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.884894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.688612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.619029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.713821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.002381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.455058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.198128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.786879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.718274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.821926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.225191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.138455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.159106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.636402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.186808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.349941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.627806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.724136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.102291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.391295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.720832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.896858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.021350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.099651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.370758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.432946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.603461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.812508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.057227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.056561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.351365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.329259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.538345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.464721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.869490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.206274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.440309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.681108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.839152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.082484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.054884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.902833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.700877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.697256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.676239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.590478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.487915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.612707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.391864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.624827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.600971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.662381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.584993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.550981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.509902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.541580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.420759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.427981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.411570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.593636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.861356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.847330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.804319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.129861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.272837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.158752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.571855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.980227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.251015] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.231338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.500719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.822092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.987976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.285182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.526746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.608768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.540538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.872681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.239586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.435786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.742479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.891652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.871206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.954995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.106525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.818433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.451645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.316305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.422138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.042040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.204581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.039541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.158331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.268799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.393555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.653752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.582883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.908780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.138156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.432928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.543168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.054428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.297776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.742549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.816671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.365147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.741292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.075192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.365770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.559218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.636763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.987207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.578712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.608910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.874442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.212832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.048508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.199254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.480246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.147250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.234108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.490952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.424466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.419518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.577522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.829415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.222416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.258974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.503795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.063443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.478845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.528637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.922943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.177138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.422479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.208847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.410683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.741979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.793166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.601866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.532267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.376921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.323578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.253134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.465307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.711261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.754387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.107902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.148146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.561289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.674300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.856512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.122231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.109858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.367939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.658035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.648411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.825016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.146469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.510506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.525857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.499069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.406404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.567378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.587163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.844864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.143585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.003277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.296391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.383966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.578968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.001560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.416146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.553238] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6546.572636] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 6546.584188] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6546.586060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.593408] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 1 previous similar message [ 6548.100887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.193111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.480703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6552.172479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.856656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.504269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.829605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.020991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.072402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.244690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.397945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.815988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.828351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.802269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.022877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.249827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.478716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.528793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6570.690582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6571.655627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6572.631552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6573.328758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.001459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6575.121876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.109127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6577.610819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6579.206289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.266332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.184670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.364987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.670476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6585.145049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6586.505741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.959100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.630480] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6589.640278] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 6589.646754] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6589.652894] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 1 previous similar message [ 6589.668435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6590.736484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.103413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.632255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6594.534160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6595.774495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6596.813701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.039718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.893885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6600.218882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6601.323802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6602.507294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6603.546859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6604.802887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6606.269060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6607.503854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6608.861328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.946876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6610.908913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6612.088333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.067823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6614.203299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6615.070455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6616.266957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6617.459044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6618.520791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6619.580922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6620.581776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6621.815882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.168271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6624.383758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6625.625033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6626.860567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6627.809972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6628.996539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6630.222789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6631.565977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6632.878107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6633.942424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6635.245135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6636.457823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6637.482124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6638.320711] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6638.327784] Lustre: Skipped 418 previous similar messages [ 6638.347405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6638.356453] Lustre: Skipped 418 previous similar messages [ 6638.359734] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6638.362338] Lustre: Skipped 418 previous similar messages [ 6638.366808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6638.377381] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6638.381040] Lustre: Skipped 418 previous similar messages [ 6639.731408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6640.857928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6642.138637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6643.531561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6644.990952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6646.025443] LustreError: 199597:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 6646.038321] LustreError: 199597:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 6640 previous similar messages [ 6646.057748] LustreError: 199597:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 6646.073435] LustreError: 199597:0:(mdt_open.c:646:mdt_finish_open()) Skipped 6640 previous similar messages [ 6646.372879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6647.574994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6648.503625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6649.549992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6650.792528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6651.736838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6652.702483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6653.744659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6654.799516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6655.962850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6657.370766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6658.773379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6660.025111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6661.447826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6662.693127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6663.606244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6664.510404] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6664.532222] LustreError: 172193:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6664.536682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6665.428076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6666.696907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6668.120683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6669.231212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6670.459832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6671.462737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6672.754528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6674.053398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6675.413985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6676.673474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6677.785812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6678.945152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6680.117923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6681.557937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6682.908549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6684.239927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6685.732672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6686.719922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6688.000514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6689.211473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6690.279894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6691.560229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6692.967959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6694.227637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6695.362134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6696.572663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6697.675785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6698.737426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6699.789946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6700.850953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6701.981598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6702.943639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6703.973260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6704.972900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6706.034762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6707.046974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6708.350770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6709.376412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6710.402168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6711.407155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6712.493233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6713.618391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6714.795596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6715.692836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6716.729695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6717.942937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6719.053719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6719.961871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6721.069608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6721.982851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6723.259731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6724.381403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6726.070773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6727.411538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6728.439255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6729.892209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6731.035196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6732.482883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6734.040283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6735.511124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6736.990045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6738.257775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6739.339074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6740.464985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6741.813602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6742.919602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6744.264096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6745.564801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6746.676138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6747.887386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6748.984804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6749.739696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6750.699331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6751.629369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6752.753794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6753.993047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6754.969667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6755.855735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6756.912933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6758.007760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6758.887735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6759.933598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6760.728362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6761.841311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6762.798781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6764.207752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6765.561082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6766.825778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6768.299228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6769.859145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6771.415480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6772.272294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6773.516670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6774.781722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6775.985101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6777.429402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6779.459715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6780.348675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6782.081134] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6783.827825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6785.047888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6785.979424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6787.443551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6788.817563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6790.214582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6791.903748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7246.032223] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255619 [ 7246.039762] LustreError: 173415:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 32367 previous similar messages [ 7246.068929] LustreError: 172168:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255619 [ 7246.076543] LustreError: 172168:0:(mdt_open.c:646:mdt_finish_open()) Skipped 32368 previous similar messages [ 7299.911735] LustreError: 182466:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 7299.915235] LustreError: 172193:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 8030.179973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.184758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.196877] Lustre: Skipped 129 previous similar messages [ 8035.298531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8035.318491] Lustre: Skipped 2 previous similar messages [ 8035.834512] Lustre: server umount lustre-OST0000 complete [ 8040.418105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8040.422956] Lustre: Skipped 1 previous similar message [ 8045.539169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.558881] Lustre: Skipped 1 previous similar message [ 8047.035643] Lustre: server umount lustre-MDT0000 complete [ 8050.664322] LustreError: 172168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.681579] LustreError: 172168:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8052.028019] LustreError: 172153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789411519 with bad export cookie 15717618503517408825 [ 8052.033795] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8052.043882] LustreError: 172153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8052.551910] Lustre: server umount lustre-MDT0001 complete [ 8067.097628] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8070.008219] Key type lgssc unregistered [ 8070.267239] LNet: 227494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8070.277658] LNetError: 227494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8070.309049] LNet: Removed LNI 192.168.201.107@tcp [ 8071.686551] Key type .llcrypt unregistered [ 8071.687735] Key type ._llcrypt unregistered [ 8100.868479] Key type ._llcrypt registered [ 8100.875180] Key type .llcrypt registered [ 8101.028923] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8102.612093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8102.696097] alg: No test for adler32 (adler32-zlib) [ 8103.938175] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 8104.160324] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 8105.847487] Key type lgssc registered [ 8107.172594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8115.793450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8115.812973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8117.332344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8122.488957] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8127.466921] LustreError: 228914:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8127.480266] LustreError: 228914:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8131.926763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8132.281672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8137.237756] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8146.619767] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8153.940403] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8162.541436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.158035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8164.224216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 8173.082929] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8184.223361] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8188.688712] LustreError: 228909:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199040032834 [ 8189.687916] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8191.911489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8191.930315] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 8191.942741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8191.964951] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8191.972198] LustreError: 228934:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 8196.014976] Lustre: lustre-OST0000: Not available for connect from 192.168.201.7@tcp (stopping) [ 8197.090257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8197.092265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8197.105663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8200.781279] Lustre: server umount lustre-OST0000 complete [ 8212.055574] LustreError: 228934:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 8237.146943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8237.491602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.938897] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8253.271171] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8262.003657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8262.029756] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8262.030819] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8263.829372] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec [ 8270.398776] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff924a48e0e000.ost_server_uuid 50 [ 8272.131988] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff924a48e0e000.ost_server_uuid in FULL state after 0 sec [ 8272.276135] LustreError: 228911:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 8272.286734] LustreError: 228911:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [ 8272.298939] LustreError: 228911:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 8277.475877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8277.491839] Lustre: Skipped 2 previous similar messages [ 8277.492815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8277.499119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8277.501138] Lustre: Skipped 2 previous similar messages [ 8279.695180] Lustre: server umount lustre-OST0000 complete [ 8285.668261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8285.675392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.684298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8290.287508] Lustre: server umount lustre-MDT0000 complete [ 8292.844290] LustreError: 228915:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8292.880225] LustreError: 228915:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8294.419235] LustreError: 229649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789411762 with bad export cookie 11422638322341948045 [ 8294.430511] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8294.434368] LustreError: 229649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8295.043668] Lustre: server umount lustre-MDT0001 complete [ 8315.762399] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 8319.108921] Key type lgssc unregistered [ 8319.394144] LNet: 234149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8319.400369] LNetError: 234149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8319.430375] LNet: Removed LNI 192.168.201.107@tcp [ 8320.481454] Key type .llcrypt unregistered [ 8320.483260] Key type ._llcrypt unregistered [ 8357.816527] Key type ._llcrypt registered [ 8357.818774] Key type .llcrypt registered [ 8357.905525] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8358.874585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8359.099084] alg: No test for adler32 (adler32-zlib) [ 8360.147223] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 8360.317992] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 8361.967753] Key type lgssc registered [ 8363.181672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8373.335801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8373.387499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.092556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.644260] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8384.520073] LustreError: 235884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8384.544545] LustreError: 235884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8389.600733] LustreError: 235885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8392.660849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.220363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8399.449651] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8403.380815] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8411.909472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8412.522767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8419.804455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 8420.822820] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8433.527930] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8433.923980] Lustre: lustre-OST0001: new disk, initializing [ 8433.928657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8433.942544] Lustre: 238195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8434.143094] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8441.021702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8441.038404] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8441.209420] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8442.964934] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8457.323293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8466.562584] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8468.412750] LustreError: 238364:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255746 [ 8468.417681] LustreError: 238364:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023255746 [ 8477.987437] LustreError: 238364:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199040032834 [ 8477.992414] LustreError: 238364:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199040032834 [ 8482.284402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8482.292717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8482.303947] Lustre: Skipped 2 previous similar messages [ 8482.315032] Lustre: Skipped 3 previous similar messages [ 8487.404724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8487.409050] Lustre: Skipped 2 previous similar messages [ 8488.670792] Lustre: server umount lustre-MDT0000 complete [ 8492.510716] LustreError: 235864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789411960 with bad export cookie 15272389802354127175 [ 8492.517085] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8492.523868] LustreError: 235864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8492.541542] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.543993] LustreError: 236869:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.548378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8492.551913] Lustre: Skipped 2 previous similar messages [ 8492.565213] LustreError: 236869:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8492.845516] Lustre: server umount lustre-MDT0001 complete [ 8499.402912] Lustre: server umount lustre-OST0000 complete [ 8504.286526] Lustre: server umount lustre-OST0001 complete [ 8529.379618] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8540.997088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.545753] LustreError: 241303:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.640030] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8546.783808] LustreError: 241304:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8547.477469] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8557.027368] LustreError: 241303:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8557.042176] LustreError: 241303:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8559.790406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.240520] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8565.499192] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8569.653313] Lustre: 242445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8577.377147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8577.961474] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8585.135176] LustreError: 242799:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.199083] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 8586.424511] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8587.187153] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 8595.425682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8601.086224] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 8601.771554] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8610.256440] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8614.424874] Lustre: 244318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8622.380143] LustreError: 241298:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199040032834 [ 8622.390763] LustreError: 241298:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199040032834 [ 8622.406240] LustreError: 241298:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 8631.779548] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8631.791194] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8631.801849] Lustre: Skipped 2 previous similar messages [ 8631.803054] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8631.810625] Lustre: Skipped 3 previous similar messages [ 8633.821892] Lustre: server umount mylustre-MDT0000 complete [ 8636.897860] LustreError: 241304:0:(ldlm_lib.c:1199:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8636.921137] LustreError: 241304:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8638.307757] LustreError: 242448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412106 with bad export cookie 15272389802354130038 [ 8638.332983] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8638.633206] Lustre: server umount mylustre-MDT0001 complete [ 8652.568543] Lustre: server umount mylustre-OST0000 complete [ 8667.007266] Lustre: server umount mylustre-OST0001 complete [ 8692.309419] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8703.693493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.282884] LustreError: 246630:0:(ldlm_lib.c:1199:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8704.408117] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8704.413690] Lustre: Skipped 1 previous similar message [ 8708.393716] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8718.309753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8723.588783] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8726.727763] Lustre: 247774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8735.188766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8742.549862] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8743.926788] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 8750.114118] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 8753.350375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8758.814140] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 8762.953764] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8774.612125] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8780.722039] Lustre: 249646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8794.602539] Lustre: tfs-MDT0000-lwp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.615053] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8794.632643] Lustre: Skipped 3 previous similar messages [ 8794.669648] Lustre: Skipped 4 previous similar messages [ 8797.893401] Lustre: server umount tfs-MDT0000 complete [ 8804.817604] LustreError: 248152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412272 with bad export cookie 15272389802354132215 [ 8804.826952] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8804.830813] LustreError: 248152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 8804.851470] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8804.860276] Lustre: tfs-MDT0001: Not available for connect from 0@lo (stopping) [ 8804.878608] Lustre: Skipped 2 previous similar messages [ 8805.188533] Lustre: server umount tfs-MDT0001 complete [ 8812.632783] Lustre: server umount tfs-OST0000 complete [ 8818.116127] Lustre: server umount tfs-OST0001 complete [ 8848.627949] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 8863.119229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8863.614246] LustreError: 251899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8863.633195] LustreError: 251899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8863.733717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8863.738343] Lustre: Skipped 3 previous similar messages [ 8869.565178] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8881.455343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8887.069269] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8890.470273] Lustre: 253041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8897.763831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8906.335314] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8907.281503] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 8908.372824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 8918.002596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8923.653606] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 8925.565523] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 8933.992446] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8938.073883] Lustre: 254911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8945.557966] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 15:00:11 (1789412411) [ 8964.079653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8964.103494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8964.135913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8964.151542] Lustre: Skipped 1 previous similar message [ 8969.637790] Lustre: server umount lustre-MDT0000 complete [ 8976.041120] LustreError: 251875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412443 with bad export cookie 15272389802354134133 [ 8976.070235] LustreError: 251875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8976.071055] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.627240] Lustre: server umount lustre-MDT0001 complete [ 8993.105309] Lustre: server umount lustre-OST0000 complete [ 9003.614586] Lustre: server umount lustre-OST0001 complete [ 9011.079533] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 9024.142597] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9066.472181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.717572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9066.755317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9066.863831] Lustre: lustre-MDT0000: new disk, initializing [ 9066.924113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 9066.927106] Lustre: Skipped 3 previous similar messages [ 9066.935384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9071.969370] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9082.358353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9082.502608] Lustre: 258112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9082.550893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9082.554798] Lustre: Skipped 1 previous similar message [ 9082.665994] Lustre: lustre-MDT0001: new disk, initializing [ 9082.856554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9082.883302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9087.827212] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9099.100898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9099.537151] Lustre: lustre-OST0000: new disk, initializing [ 9099.547945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9099.572855] Lustre: 259693:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9101.231906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9101.248842] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9101.467071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9107.031322] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9118.699882] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9118.929945] Lustre: lustre-OST0001: new disk, initializing [ 9118.934322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9118.938113] Lustre: 260567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9120.746346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9120.759710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9120.872515] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9127.551989] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9146.399804] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 15:03:31 (1789412611) [ 9162.721501] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 15:03:48 (1789412628) [ 9182.184859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9182.214143] Lustre: Skipped 6 previous similar messages [ 9182.215942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9182.227406] Lustre: Skipped 8 previous similar messages [ 9185.413309] Lustre: server umount lustre-OST0000 complete [ 9187.301528] LustreError: 259679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9187.341886] LustreError: 259679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [ 9197.350802] Lustre: server umount lustre-MDT0000 complete [ 9202.715979] LustreError: 258103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412670 with bad export cookie 15272389802354135911 [ 9202.723897] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9202.731615] LustreError: 258103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9209.716595] Lustre: server umount lustre-MDT0001 complete [ 9222.216572] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9301.154451] Lustre: server umount lustre-OST0001 complete [ 9310.233297] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [ 9334.055995] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9379.873978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9380.200916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9380.264810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9380.362715] Lustre: lustre-MDT0000: new disk, initializing [ 9380.467305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9380.469579] Lustre: Skipped 3 previous similar messages [ 9380.492652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9386.835575] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9401.359478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9401.598271] Lustre: 265427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9401.712705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9401.716980] Lustre: Skipped 1 previous similar message [ 9401.839597] Lustre: lustre-MDT0001: new disk, initializing [ 9402.104571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9402.127821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9407.821096] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9419.873804] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9427.975714] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9436.778735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9437.239632] Lustre: lustre-OST0000: new disk, initializing [ 9437.248281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9437.260122] Lustre: 267398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9438.616310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9438.629027] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9438.832658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9445.763986] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9459.311607] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9462.086836] LustreError: 267405:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [ 9462.089846] LustreError: 267405:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [ 9462.095694] LustreError: 267405:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [ 9469.408332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9469.412898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9469.428168] LustreError: Skipped 1 previous similar message [ 9469.430763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9469.430774] Lustre: Skipped 4 previous similar messages [ 9469.463051] Lustre: Skipped 5 previous similar messages [ 9474.190736] Lustre: server umount lustre-OST0000 complete [ 9485.754671] Lustre: server umount lustre-MDT0000 complete [ 9490.641348] LustreError: 266225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789412958 with bad export cookie 15272389802354137381 [ 9490.651324] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9490.659249] LustreError: 266225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9491.134446] Lustre: server umount lustre-MDT0001 complete [ 9509.677071] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9513.542100] Key type lgssc unregistered [ 9514.236854] LNet: 269179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9514.241650] LNetError: 269179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9514.263366] LNet: Removed LNI 192.168.201.107@tcp [ 9515.810243] Key type .llcrypt unregistered [ 9515.811868] Key type ._llcrypt unregistered [ 9551.385950] Key type ._llcrypt registered [ 9551.387336] Key type .llcrypt registered [ 9551.533332] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9552.761890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9552.927985] alg: No test for adler32 (adler32-zlib) [ 9554.186413] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 9554.509985] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 9556.320643] Key type lgssc registered [ 9557.739788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9569.643055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9569.720028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.607818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9579.326976] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9582.062703] LustreError: 270641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9582.082363] LustreError: 270641:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9587.200642] LustreError: 270642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.287983] LustreError: 270641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.874468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9593.337351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9599.781973] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9609.339429] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9618.043658] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9627.527297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9627.937717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9636.757690] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9642.296815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 9648.179853] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9649.699751] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9652.704081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9652.709395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.724565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9652.756745] Lustre: Skipped 1 previous similar message [ 9657.833048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9657.849383] Lustre: Skipped 2 previous similar messages [ 9658.496618] Lustre: server umount lustre-OST0000 complete [ 9665.001360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9665.007751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9665.040724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9668.067182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9668.068891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9669.816333] Lustre: server umount lustre-MDT0000 complete [ 9673.184667] LustreError: 272678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9673.215487] LustreError: 272678:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9674.182237] LustreError: 273172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413141 with bad export cookie 1821099805466539047 [ 9674.195748] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9674.201534] LustreError: 273172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9674.947847] Lustre: server umount lustre-MDT0001 complete [ 9689.963398] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9693.887887] Key type lgssc unregistered [ 9694.179884] LNet: 274228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9694.189387] LNetError: 274228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9695.204844] LNet: Removed LNI 192.168.201.107@tcp [ 9696.203346] Key type .llcrypt unregistered [ 9696.205436] Key type ._llcrypt unregistered [ 9719.980544] Key type ._llcrypt registered [ 9719.983314] Key type .llcrypt registered [ 9720.102244] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9721.191370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9721.250150] alg: No test for adler32 (adler32-zlib) [ 9723.013446] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 9723.598148] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 9725.447401] Key type lgssc registered [ 9727.081651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9742.201602] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9742.228623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9757.855490] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9762.976117] LustreError: 275423:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.107@tcp: failed processing log, type 4: rc = -110 [ 9789.663239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9797.053213] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [ 9805.356398] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9808.190849] Lustre: server umount lustre-OST0000 complete [ 9827.536590] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [ 9831.771167] Key type lgssc unregistered [ 9832.089124] LNet: 277054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9832.105580] LNetError: 277054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9832.121984] LNet: Removed LNI 192.168.201.107@tcp [ 9833.254304] Key type .llcrypt unregistered [ 9833.258681] Key type ._llcrypt unregistered [ 9880.372525] Key type ._llcrypt registered [ 9880.378292] Key type .llcrypt registered [ 9880.564593] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [ 9881.934756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9881.968737] alg: No test for adler32 (adler32-zlib) [ 9882.947929] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [ 9883.119789] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [ 9884.759426] Key type lgssc registered [ 9886.504752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9903.871350] loop0: detected capacity change from 0 to 204800000 [ 9904.069519] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9904.085408] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9904.559767] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9905.896498] loop0: detected capacity change from 0 to 204800000 [ 9906.189537] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9906.204792] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9906.742187] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9908.003706] loop0: detected capacity change from 0 to 204800000 [ 9908.231334] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9908.243058] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9908.361701] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9909.901461] loop0: detected capacity change from 0 to 204800000 [ 9910.165830] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9910.195594] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9910.346208] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9911.612431] loop0: detected capacity change from 0 to 204800000 [ 9911.663935] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9916.663369] loop0: detected capacity change from 0 to 204800000 [ 9916.690746] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9920.903065] loop0: detected capacity change from 0 to 204800000 [ 9920.937470] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9925.858193] loop0: detected capacity change from 0 to 204800000 [ 9925.890749] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9930.284558] loop0: detected capacity change from 0 to 204800000 [ 9930.430875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9930.517462] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9930.678632] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9930.708524] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9930.728346] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9930.755061] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9930.789548] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9930.799919] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9930.817086] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9930.827500] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9930.836046] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9930.859365] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9930.875871] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9930.904431] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9930.938609] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9930.955507] Lustre: 280463:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9938.024694] Lustre: server umount MGS complete [ 9939.168559] loop0: detected capacity change from 0 to 204800000 [ 9939.247748] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9939.902630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9939.958109] Lustre: lustre-MDT0000: Aborting client recovery [ 9940.995437] loop1: detected capacity change from 0 to 204800000 [ 9941.202048] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9941.349294] Lustre: 280894:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9941.363633] Lustre: 280894:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9941.378273] Lustre: 280894:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9941.388543] Lustre: 280894:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9941.411816] Lustre: 280894:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9941.426778] Lustre: 280894:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9941.443780] Lustre: 280894:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9941.477700] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9941.662669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9941.727795] Lustre: lustre-MDT0001: Aborting client recovery [ 9942.864429] loop2: detected capacity change from 0 to 204800000 [ 9943.045482] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9943.120477] Lustre: 280974:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9943.137541] Lustre: 280974:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9943.152943] Lustre: 280974:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9943.185503] Lustre: 280974:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9943.205331] Lustre: 280974:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9943.227327] Lustre: 280974:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9943.250565] Lustre: 280974:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9943.373598] Lustre: Found index 0 for lustre-OST0000, updating log [ 9943.416808] Lustre: 280992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9943.484244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9943.503484] Lustre: lustre-OST0000: Aborting recovery [ 9944.634633] loop3: detected capacity change from 0 to 204800000 [ 9944.858135] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9944.976199] Lustre: 281057:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9944.992736] Lustre: 281057:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9945.006892] Lustre: 281057:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9945.022914] Lustre: 281057:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9945.042768] Lustre: 281057:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9945.068792] Lustre: 281057:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9945.083961] Lustre: 281057:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9945.124634] Lustre: Found index 1 for lustre-OST0001, updating log [ 9945.185643] Lustre: 281067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9945.302843] Lustre: lustre-OST0001: Aborting recovery [ 9949.366640] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9949.375187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9949.415881] LustreError: 280838:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9949.428476] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9949.475691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9951.407776] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9951.423997] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9954.794874] LustreError: 280805:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 13102us: rc = -2 [ 9955.241655] LustreError: 280804:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023256130 [ 9955.254697] LustreError: 280804:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199023256130 [ 9955.294364] LustreError: 280803:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 646us: rc = -2 [ 9955.319585] LustreError: 280803:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 13 previous similar messages [ 9956.031573] LustreError: 280804:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199602070594 [ 9956.042150] LustreError: 280804:0:(mdt_open.c:646:mdt_finish_open()) Skipped 3 previous similar messages [ 9956.090203] LustreError: 280803:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199023255553 [ 9956.102032] LustreError: 280803:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 3 previous similar messages [ 9956.297126] LustreError: 280804:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 2652us: rc = -2 [ 9956.326396] LustreError: 280804:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 21 previous similar messages [ 9972.208809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9972.210897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9972.230300] Lustre: Skipped 1 previous similar message [ 9972.250289] Lustre: Skipped 3 previous similar messages [ 9976.443870] Lustre: server umount lustre-MDT0000 complete [ 9977.318215] LustreError: 280803:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9977.351815] LustreError: 280803:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9977.508820] LustreError: 280788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413445 with bad export cookie 3395990377031141488 [ 9977.528888] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9977.543334] LustreError: 280788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9982.435150] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.435535] LustreError: 280802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9982.447648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9982.448607] Lustre: Skipped 3 previous similar messages [ 9982.510194] LustreError: 280802:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9984.241145] Lustre: server umount lustre-MDT0001 complete [ 9992.017554] Lustre: server umount lustre-OST0000 complete [10000.223828] Lustre: server umount lustre-OST0001 complete [10013.135215] Lustre: DEBUG MARKER: == conf-sanity test 108c: mkfs.lustre --mmp option enables MMP feature ========================================================== 15:17:58 (1789413478) [10037.448668] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [10082.413673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10082.796216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10082.833819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10082.956694] Lustre: lustre-MDT0000: new disk, initializing [10083.079344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10083.082290] Lustre: Skipped 1 previous similar message [10083.096831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10088.171306] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10096.387535] Lustre: server umount lustre-MDT0000 complete [10121.917061] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 15:19:47 (1789413587) [10152.170247] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [10161.310724] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [10203.818219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10204.199970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10204.246261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10204.253075] Lustre: Skipped 1 previous similar message [10204.327248] Lustre: lustre-MDT0000: new disk, initializing [10204.393448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10204.409132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10209.751706] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10222.292344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10222.493755] Lustre: 288174:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10222.601280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10222.605552] Lustre: Skipped 1 previous similar message [10222.693314] Lustre: lustre-MDT0001: new disk, initializing [10222.848757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10222.934087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10222.958592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10228.192781] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10238.921798] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10245.885607] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10254.905106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10255.152123] Lustre: lustre-OST0000: new disk, initializing [10255.155495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10255.160155] Lustre: 290144:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10255.248488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10261.627578] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10263.103528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10263.113216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10263.262394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10273.096209] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10279.377882] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10292.046887] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10300.451975] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10345.443574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10345.454133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10345.456803] LustreError: Skipped 1 previous similar message [10345.457494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10345.457499] Lustre: Skipped 1 previous similar message [10345.481386] Lustre: Skipped 1 previous similar message [10350.563296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10350.575079] Lustre: Skipped 2 previous similar messages [10355.712750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10355.730380] Lustre: Skipped 1 previous similar message [10359.265069] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10359.517981] Lustre: server umount lustre-OST0000 complete [10365.923236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10365.930193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10365.957908] Lustre: Skipped 1 previous similar message [10365.977021] Lustre: Skipped 1 previous similar message [10372.107946] Lustre: server umount lustre-MDT0000 complete [10376.162283] LustreError: 288182:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10376.183632] LustreError: 288182:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10377.186278] LustreError: 288168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413844 with bad export cookie 3395990377031144554 [10377.186977] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10377.202608] LustreError: 288168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10377.680978] Lustre: server umount lustre-MDT0001 complete [10388.296965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10393.059475] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10398.423363] Lustre: server umount MGS complete [10412.141650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10412.749035] LustreError: 294062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10412.797768] LustreError: 294075:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10412.944341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10418.148063] LustreError: 294063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10420.881162] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10428.384842] LustreError: 294062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10428.407633] LustreError: 294062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10435.503472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10441.109307] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10449.676694] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10455.838735] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10463.024211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10470.746881] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10478.313117] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10509.797500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10509.804447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10509.825786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10509.836050] Lustre: Skipped 3 previous similar messages [10511.025536] Lustre: server umount lustre-OST0000 complete [10517.992319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10518.009066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10518.032141] Lustre: Skipped 1 previous similar message [10521.929426] Lustre: server umount lustre-MDT0000 complete [10525.167409] LustreError: 294062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10525.205702] LustreError: 294062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10526.356743] LustreError: 296788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789413994 with bad export cookie 3395990377031146395 [10526.366204] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10526.372354] LustreError: 296788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10527.123187] Lustre: server umount lustre-MDT0001 complete [10539.091040] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [10542.245835] Key type lgssc unregistered [10542.666271] LNet: 298004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10542.690451] LNetError: 298004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10543.721525] LNet: Removed LNI 192.168.201.107@tcp [10545.098635] Key type .llcrypt unregistered [10545.102388] Key type ._llcrypt unregistered [10594.699827] Key type ._llcrypt registered [10594.707418] Key type .llcrypt registered [10594.934558] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [10613.408211] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [10615.294438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10615.367540] alg: No test for adler32 (adler32-zlib) [10616.353988] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [10616.524246] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [10618.177763] Key type lgssc registered [10619.467341] Lustre: Echo OBD driver; http://www.lustre.org/ [10661.629964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10661.673724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10663.073338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10663.121381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10663.200290] Lustre: lustre-MDT0000: new disk, initializing [10663.268055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10663.283108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10667.153758] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10678.876655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.054236] Lustre: 301964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10679.098181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10679.101398] Lustre: Skipped 1 previous similar message [10679.177947] Lustre: lustre-MDT0001: new disk, initializing [10679.317863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10679.373042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10679.401192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10684.168802] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10694.930205] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10702.937514] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10713.030551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10713.360864] Lustre: lustre-OST0000: new disk, initializing [10713.370283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10713.385836] Lustre: 303937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10713.576789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10720.958766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10720.990658] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10721.367611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10721.970137] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10732.590259] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10739.979834] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10752.383585] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10759.686925] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10766.914743] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [10808.293187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10808.296330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10808.300128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10808.306749] LustreError: Skipped 1 previous similar message [10808.325840] Lustre: Skipped 1 previous similar message [10812.408840] Lustre: server umount lustre-OST0000 complete [10817.512942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10817.523904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10817.554745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10817.566117] Lustre: Skipped 1 previous similar message [10818.531349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10818.561691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10818.576728] Lustre: Skipped 1 previous similar message [10823.650296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10823.659561] Lustre: Skipped 1 previous similar message [10828.780831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10828.802727] Lustre: Skipped 1 previous similar message [10831.327148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10831.694727] Lustre: server umount lustre-MDT0000 complete [10835.985315] LustreError: 301955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789414303 with bad export cookie 10519121088178398137 [10835.995764] LustreError: 301955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10835.997030] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10836.515804] Lustre: server umount lustre-MDT0001 complete [10846.638744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.533621] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10857.128195] Lustre: server umount MGS complete [10871.567744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10872.210422] LustreError: 307904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.235730] LustreError: 307904:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10872.251710] LustreError: 307917:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10872.326554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10877.410312] LustreError: 307905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10877.970188] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10881.507227] LustreError: 307904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10886.626986] LustreError: 307905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10887.267823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10887.687698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10893.178604] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10902.162730] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10908.886584] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10916.843134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10917.207423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10923.833535] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [10931.297917] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10972.641101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10972.650233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10972.661347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10972.662967] Lustre: Skipped 1 previous similar message [10972.673704] Lustre: Skipped 1 previous similar message [10977.428715] Lustre: server umount lustre-OST0000 complete [10982.882206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10982.896337] Lustre: Skipped 1 previous similar message [10988.656521] Lustre: server umount lustre-MDT0000 complete [10992.882726] LustreError: 307885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789414460 with bad export cookie 10519121088178400090 [10992.893131] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10992.897301] LustreError: 307885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10993.193905] Lustre: server umount lustre-MDT0001 complete [11005.928509] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [11010.041944] Key type lgssc unregistered [11010.434947] LNet: 311873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11010.448673] LNetError: 311873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11010.463844] LNet: Removed LNI 192.168.201.107@tcp [11011.688816] Key type .llcrypt unregistered [11011.691306] Key type ._llcrypt unregistered [11035.142858] Key type ._llcrypt registered [11035.145388] Key type .llcrypt registered [11035.257479] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [11051.120656] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [11052.514268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11052.585788] alg: No test for adler32 (adler32-zlib) [11053.657309] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [11053.873212] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [11055.535241] Key type lgssc registered [11056.766364] Lustre: Echo OBD driver; http://www.lustre.org/ [11083.503269] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [11098.157924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11098.189385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11099.436414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11099.478174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11099.568304] Lustre: lustre-MDT0000: new disk, initializing [11099.700345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11099.735932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11105.171604] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [11116.317348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11116.583136] Lustre: lustre-OST0000: new disk, initializing [11116.587729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11116.591651] Lustre: Skipped 1 previous similar message [11116.596318] Lustre: 315921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11116.670164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11124.138540] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [11125.894935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11125.910868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11125.947026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11133.465693] LustreError: 315122:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613190210 [11133.473837] LustreError: 315122:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2207613190210 [11994.764923] Lustre: server umount lustre-MDT0000 complete [11998.260813] LustreError: 315108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789415466 with bad export cookie 15333331849258015719 [11998.278434] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12008.589870] Lustre: server umount lustre-OST0000 complete [12035.263358] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [12047.186872] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [12097.316174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12097.558591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12097.604766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12097.684433] Lustre: lustre-MDT0000: new disk, initializing [12097.787101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12097.821387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12102.871203] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12114.232297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12114.376243] Lustre: 320264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12114.446277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12114.454025] Lustre: Skipped 1 previous similar message [12114.565427] Lustre: lustre-MDT0001: new disk, initializing [12114.697876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12114.754936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12114.767036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12120.548169] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12130.670899] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12138.378433] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12147.477648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12147.820074] Lustre: lustre-OST0000: new disk, initializing [12147.830605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12147.841245] Lustre: 322236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12147.966427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12155.663925] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12156.977963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12156.988599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12157.117234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12167.474880] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12174.199887] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12174.373670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12180.821519] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12181.279775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12188.134621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12188.149536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12188.154931] Lustre: Skipped 1 previous similar message [12188.174585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12188.192073] LustreError: Skipped 1 previous similar message [12190.490923] Lustre: server umount lustre-OST0000 complete [12196.848738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12196.859906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12196.880292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12196.890416] Lustre: Skipped 1 previous similar message [12198.381652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12198.382888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12201.111702] Lustre: server umount lustre-MDT0000 complete [12205.705478] LustreError: 322234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789415673 with bad export cookie 15333331849260956538 [12205.714983] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12205.721523] LustreError: 322234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12206.088405] Lustre: server umount lustre-MDT0001 complete [12209.977551] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [12211.920343] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 15:54:37 (1789415677) [12219.487332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12219.861089] LustreError: 324550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12219.882614] LustreError: 324550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12219.920956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12224.992102] LustreError: 324551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12225.650735] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12230.119779] LustreError: 324550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12235.237257] LustreError: 324551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12238.161812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.848987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12245.814607] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12254.705574] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12262.700947] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12270.515651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12271.037865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12276.914990] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12285.259942] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12293.752696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12293.970224] Lustre: lustre-OST0001: new disk, initializing [12293.976921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12293.983333] Lustre: 327388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12294.096870] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12302.550303] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12303.512387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12303.534090] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12303.601136] LustreError: 327779:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12304.689873] LustreError: 327787:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [12304.720994] LustreError: 327787:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [12305.762636] LustreError: 327779:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12307.889385] LustreError: 327787:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [12307.915474] LustreError: 327787:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [12312.048924] LustreError: 327779:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12312.077213] LustreError: 327779:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [12316.208083] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12316.566965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12320.298327] LustreError: 327779:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [12320.311576] LustreError: 327779:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 14 previous similar messages [12324.549597] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff924a45377800.ost_server_uuid 50 [12326.285987] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff924a45377800.ost_server_uuid in FULL state after 0 sec [12332.900130] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff924a45377800.ost_server_uuid 50 [12334.762986] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff924a45377800.ost_server_uuid in IDLE state after 0 sec [12335.036847] LustreError: 324547:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199040032834 [12335.040045] LustreError: 324547:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [12335.049622] LustreError: 324547:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199040032834 [12335.061717] LustreError: 324547:0:(mdt_open.c:646:mdt_finish_open()) Skipped 1 previous similar message [12336.891358] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12347.500978] LustreError: 325480:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2199040032834 [12347.505066] LustreError: 325480:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 1 previous similar message [12347.509129] LustreError: 325480:0:(mdt_open.c:646:mdt_finish_open()) 4 open_flags 2199040032834 [12347.516943] LustreError: 325480:0:(mdt_open.c:646:mdt_finish_open()) Skipped 3 previous similar messages [12352.483539] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12352.483784] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12352.494505] LustreError: Skipped 1 previous similar message [12352.495657] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12352.495667] Lustre: Skipped 1 previous similar message [12352.547564] Lustre: Skipped 1 previous similar message [12356.491854] Lustre: server umount lustre-OST0001 complete [12360.085511] LustreError: 327182:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12365.311462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12365.312730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12365.335913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12365.361085] Lustre: Skipped 5 previous similar messages [12370.408947] LustreError: 326372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12370.427980] LustreError: 326372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12371.651546] Lustre: server umount lustre-OST0000 complete [12377.056311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12377.061975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12377.078226] Lustre: Skipped 1 previous similar message [12377.090775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12377.107734] Lustre: Skipped 2 previous similar messages [12390.373454] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12390.642314] Lustre: server umount lustre-MDT0000 complete [12390.911712] LustreError: 324550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12390.924030] LustreError: 324550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12394.560432] LustreError: 324531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789415862 with bad export cookie 15333331849260957420 [12394.567679] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12394.574195] LustreError: 324531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12394.812888] Lustre: server umount lustre-MDT0001 complete [12406.894454] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [12409.674347] Key type lgssc unregistered [12409.906567] LNet: 330081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12409.909698] LNetError: 330081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12409.922398] LNet: Removed LNI 192.168.201.107@tcp [12410.816570] Key type .llcrypt unregistered [12410.818568] Key type ._llcrypt unregistered [12439.077794] Key type ._llcrypt registered [12439.080030] Key type .llcrypt registered [12439.159363] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [12440.531476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12440.561229] alg: No test for adler32 (adler32-zlib) [12441.553873] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [12441.777972] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [12443.456801] Key type lgssc registered [12444.658872] Lustre: Echo OBD driver; http://www.lustre.org/ [12452.966971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12452.994321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12454.486278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12459.068424] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12464.638322] LustreError: 331500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12464.654715] LustreError: 331500:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12467.747154] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12469.735267] LustreError: 331501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12474.739871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12474.850168] LustreError: 331500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.191660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12480.957596] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12490.832469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12491.272022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12497.987516] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12499.432541] LustreError: 333181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12499.451767] LustreError: 333181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12499.469853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [12504.551888] LustreError: 333183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12504.575649] LustreError: 333183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12506.455436] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12513.382221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12513.580119] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12518.955349] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [12521.206346] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12530.695341] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12539.373907] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [12539.625431] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12542.251815] LustreError: 331495:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [12542.789930] LustreError: 331496:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [12542.802829] LustreError: 331496:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 13 previous similar messages [12543.832495] LustreError: 331495:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [12543.851786] LustreError: 331495:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 23 previous similar messages [12545.852512] LustreError: 331496:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [12545.860974] LustreError: 331496:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 67 previous similar messages [12549.859518] LustreError: 334854:0:(mdt_open.c:1779:mdt_reint_open()) 2 open_flags 2207613255681 [12549.871282] LustreError: 334854:0:(mdt_open.c:1779:mdt_reint_open()) Skipped 97 previous similar messages [12667.392494] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12667.400245] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12667.421556] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12667.421878] Lustre: Skipped 1 previous similar message [12667.444751] Lustre: Skipped 1 previous similar message [12669.764945] Lustre: server umount lustre-OST0001 complete [12672.482216] LustreError: 333614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12672.513991] LustreError: 333614:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12677.608585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12677.609863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12677.625370] Lustre: Skipped 1 previous similar message [12682.724180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12682.739985] Lustre: Skipped 2 previous similar messages [12683.931446] Lustre: server umount lustre-OST0000 complete [12690.400429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12690.405528] LustreError: Skipped 1 previous similar message [12690.408812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12690.425354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12692.963064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12696.495344] Lustre: server umount lustre-MDT0000 complete [12698.085699] LustreError: 331500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12698.111060] LustreError: 331500:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12702.211457] LustreError: 331482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416169 with bad export cookie 10718505496644673548 [12702.223521] LustreError: 331482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12702.231753] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12702.663362] Lustre: server umount lustre-MDT0001 complete [12714.767152] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [12717.056633] Key type lgssc unregistered [12717.291355] LNet: 336461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12717.297425] LNetError: 336461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12717.309669] LNet: Removed LNI 192.168.201.107@tcp [12718.183410] Key type .llcrypt unregistered [12718.187273] Key type ._llcrypt unregistered [12773.848654] Key type ._llcrypt registered [12773.850090] Key type .llcrypt registered [12774.028703] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [12775.123489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12775.260956] alg: No test for adler32 (adler32-zlib) [12776.674787] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [12776.861267] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [12778.529492] Key type lgssc registered [12781.037487] Lustre: Echo OBD driver; http://www.lustre.org/ [12794.123879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12794.141262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12795.971499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12801.383083] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12806.115079] LustreError: 338752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12806.133359] LustreError: 338752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12811.232317] LustreError: 338751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12813.568215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12814.342466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12820.314250] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12823.409224] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12831.779195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12832.359904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12838.479222] LustreError: 340239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.573608] LustreError: 340239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.591331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12840.608299] LustreError: 340239:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12840.981321] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12846.051255] LustreError: 340492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12846.067484] LustreError: 340492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12851.806663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12852.052425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12857.373965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12860.436030] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12872.964991] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12882.470733] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12901.344605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12901.358780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12901.389464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12903.399485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12903.403446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12903.427417] Lustre: Skipped 2 previous similar messages [12904.767300] Lustre: server umount lustre-MDT0000 complete [12908.519747] LustreError: 338751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12908.539265] LustreError: 338751:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12909.402089] LustreError: 338731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416377 with bad export cookie 9144584489015045876 [12909.411461] LustreError: 338731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12909.413522] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12909.977184] Lustre: server umount lustre-MDT0001 complete [12924.563957] Lustre: server umount lustre-OST0000 complete [12929.505198] Lustre: 338013:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789416381/real 1789416381] req@ffff9cef55b81880 x1876338927543680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789416397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12929.553515] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12929.832440] Lustre: server umount lustre-OST0001 complete [12947.827694] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [12960.722038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12961.180042] LustreError: 344015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12961.261808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12966.700203] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12976.409712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12976.869854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12981.871524] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [12985.264809] Lustre: 345157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12992.790362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12993.305371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13000.690402] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13001.510956] LustreError: 345512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13001.529078] LustreError: 345512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13005.622399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [13012.536935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13018.672787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [13021.795690] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13031.414635] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13036.272378] Lustre: 347027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [13043.174675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13043.182413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13043.192871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13043.197381] Lustre: Skipped 3 previous similar messages [13047.189151] Lustre: server umount lustre-MDT0000 complete [13051.233330] LustreError: 343995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416518 with bad export cookie 9144584489015047577 [13051.257690] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13051.725449] Lustre: server umount lustre-MDT0001 complete [13066.473755] Lustre: server umount lustre-OST0000 complete [13079.814533] Lustre: server umount lustre-OST0001 complete [13086.092335] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [13087.693461] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [13089.402202] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 16:09:15 (1789416555) [13119.205131] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [13132.301778] loop0: detected capacity change from 0 to 1099511627776 [13133.204742] SGI XFS with ACLs, security attributes, quota, no debug enabled [13133.209346] *** VALIDATE xfs *** [13133.220630] XFS (loop0): Mounting V5 Filesystem [13133.246154] XFS (loop0): Ending clean mount [13135.637284] loop1: detected capacity change from 0 to 18691697672192 [13261.517963] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [13271.504025] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [13312.108441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13312.378330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13312.401443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13312.456113] Lustre: lustre-MDT0000: new disk, initializing [13312.520577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13312.524360] Lustre: Skipped 1 previous similar message [13312.535871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13318.600281] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13332.157411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13332.355295] Lustre: 352173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13332.431367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13332.438977] Lustre: Skipped 1 previous similar message [13332.512953] Lustre: lustre-MDT0001: new disk, initializing [13332.636877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13332.663400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13338.111532] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13348.482604] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13355.479407] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13363.864976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13364.337133] Lustre: lustre-OST0000: new disk, initializing [13364.346892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13364.360786] Lustre: 354142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13365.640237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13365.661723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13365.814564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13371.881992] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13381.715690] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13387.019154] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13387.263603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13391.419018] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13391.559192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13396.452407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13396.452857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13396.464247] LustreError: Skipped 1 previous similar message [13396.466280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13396.476381] Lustre: Skipped 4 previous similar messages [13396.493637] Lustre: Skipped 5 previous similar messages [13399.704315] Lustre: server umount lustre-OST0000 complete [13404.128871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13404.137213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13406.689469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13406.703579] Lustre: Skipped 1 previous similar message [13410.080027] Lustre: server umount lustre-MDT0000 complete [13411.813087] LustreError: 352179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13411.827269] LustreError: 352179:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [13414.429608] LustreError: 352163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416882 with bad export cookie 9144584489015049131 [13414.447459] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13414.453305] LustreError: 352163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13414.877464] Lustre: server umount lustre-MDT0001 complete [13424.032299] XFS (loop0): Unmounting Filesystem [13429.451710] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 16:14:54 (1789416894) [13437.972886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13438.618242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13438.621438] Lustre: Skipped 2 previous similar messages [13443.360880] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13452.238561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13457.158295] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13465.274106] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13471.596207] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13479.471700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13488.370475] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13496.871604] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13508.070331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13508.072678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13508.091643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13508.108019] Lustre: Skipped 2 previous similar messages [13513.921431] Lustre: server umount lustre-OST0000 complete [13524.610890] Lustre: server umount lustre-MDT0000 complete [13528.487423] LustreError: 356682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789416996 with bad export cookie 9144584489015050013 [13528.498046] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13528.505198] LustreError: 356682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13528.866920] Lustre: server umount lustre-MDT0001 complete [13542.365308] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [13546.094429] Key type lgssc unregistered [13546.439680] LNet: 360329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13546.451040] LNetError: 360329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13546.466803] LNet: Removed LNI 192.168.201.107@tcp [13547.449188] Key type .llcrypt unregistered [13547.452188] Key type ._llcrypt unregistered [13579.819381] Key type ._llcrypt registered [13579.822235] Key type .llcrypt registered [13579.924330] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [13581.271309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13581.387091] alg: No test for adler32 (adler32-zlib) [13582.543086] Lustre: Lustre: Build Version: 2.17.58_39_gb3cb314 [13582.877099] LNet: Added LNI 192.168.201.107@tcp [8/256/0/180] [13584.583241] Key type lgssc registered [13585.581841] Lustre: Echo OBD driver; http://www.lustre.org/ [13594.394495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13594.438841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13595.956264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13600.084947] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13605.354906] LustreError: 361895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13605.379371] LustreError: 361895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13610.472925] LustreError: 361896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13614.139287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13614.982731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13626.501277] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13637.703854] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13644.650665] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13652.399509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13652.774838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13660.326616] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13668.391929] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13675.458654] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.7@tcp (stopping) [13676.522454] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13676.538589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13676.544200] Lustre: Skipped 1 previous similar message [13676.562389] Lustre: Skipped 1 previous similar message [13680.536147] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.7@tcp (stopping) [13682.055285] Lustre: server umount lustre-MDT0001 complete [13685.672242] LustreError: 361891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13690.769866] LustreError: 361892:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13690.789776] LustreError: 361892:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13695.895883] LustreError: 361892:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13695.935527] LustreError: 361892:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13698.552566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13698.642752] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13698.648232] Lustre: Found index 1 for lustre-MDT0001, updating log [13698.679755] Lustre: 361883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13699.031695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13704.183699] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13704.683116] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13706.244809] LustreError: 365420:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13706.257042] LustreError: 365420:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13706.275167] LustreError: 361888:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13706.407413] LustreError: 365429:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.107@tcp: cfg command failed: rc = -17 [13706.420216] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13706.420216] [13719.540129] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13719.551874] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13719.575216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13719.575227] Lustre: Skipped 2 previous similar messages [13719.598852] Lustre: Skipped 1 previous similar message [13723.200784] Lustre: server umount lustre-MDT0001 complete [13724.646404] LustreError: 362714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13724.660119] LustreError: 362714:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13737.027962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13737.083326] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13737.087672] Lustre: Found index 1 for lustre-MDT0001, updating log [13737.131522] Lustre: 361885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13737.387512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13742.330579] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [13742.597304] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13746.691757] LustreError: 366574:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13746.701215] LustreError: 366574:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13746.709143] LustreError: 361888:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13746.719868] LustreError: 361888:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13746.809628] LustreError: 366577:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13746.825103] LustreError: 366577:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13746.835593] LustreError: 366577:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.107@tcp: cfg command failed: rc = -17 [13746.854701] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.107@tcp [13746.854701] [14049.215379] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.7@tcp (stopping) [14049.230423] Lustre: Skipped 2 previous similar messages [14049.765733] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14049.778575] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14049.818757] Lustre: Skipped 1 previous similar message [14055.536663] Lustre: server umount lustre-MDT0001 complete [14059.404327] LustreError: 361891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14059.455842] LustreError: 361891:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [14069.107453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14069.302912] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14069.312350] Lustre: Found index 1 for lustre-MDT0001, updating log [14069.401630] Lustre: 361883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14069.783668] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [14074.820649] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [14074.870317] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14079.468724] LustreError: 374351:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [14079.487301] LustreError: 374351:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [14079.506988] LustreError: 361888:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [14079.518135] LustreError: 361888:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [14079.611081] LustreError: 374354:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [14079.638542] LustreError: 374354:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.107@tcp: cfg command failed: rc = -114 [14079.645293] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [14079.645293] [14397.414783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14397.440249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14397.445467] Lustre: Skipped 2 previous similar messages [14397.452635] Lustre: Skipped 6 previous similar messages [14403.151958] Lustre: server umount lustre-MDT0000 complete [14407.657843] LustreError: 361891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14407.690238] LustreError: 361891:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [14408.247828] LustreError: 373935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789417875 with bad export cookie 15014753170235081920 [14408.258979] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14408.266856] LustreError: 373935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14412.768835] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14413.801218] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14413.814168] Lustre: Skipped 4 previous similar messages [14414.955940] Lustre: server umount lustre-MDT0001 complete [14420.218574] Lustre: server umount lustre-OST0000 complete [14430.781671] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [14444.278654] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [14485.779392] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 16:32:31 (1789417951) [14492.775470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14493.019836] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14493.054726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14493.132342] Lustre: lustre-MDT0000: new disk, initializing [14493.212307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14493.230259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14498.063109] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [14508.727124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14508.918153] Lustre: 384236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14509.036816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14509.043981] Lustre: Skipped 1 previous similar message [14509.123636] Lustre: lustre-MDT0001: new disk, initializing [14509.271579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14509.324373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14509.342239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14514.144000] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [14525.165596] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14531.502794] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14539.373655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14539.787657] Lustre: lustre-OST0000: new disk, initializing [14539.792281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14539.802932] Lustre: 386202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14539.958047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14546.936424] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_default_debug -1 all [14548.595208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14548.606422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14548.737956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14558.111678] Lustre: DEBUG MARKER: oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14565.858983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14565.867846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14565.889559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14567.846852] Lustre: server umount lustre-MDT0000 complete [14569.457440] LustreError: 384246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14569.498499] LustreError: 384246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14571.780768] LustreError: 385041:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [14571.790168] LustreError: 384229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789418039 with bad export cookie 15014753170235083873 [14571.803508] LustreError: MGC192.168.201.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14571.813596] LustreError: 384229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14572.371647] Lustre: server umount lustre-MDT0001 complete [14587.071603] Lustre: server umount lustre-OST0000 complete [14619.423026] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing set_hostid [14632.168571] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing load_modules_local [14671.053797] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14391 sec ========= 16:35:36 (1789418136) [14673.198511] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:35:38 (1789418138) === [14677.248377] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:35:43 (1789418143) === [14725.734343] Lustre: DEBUG MARKER: oleg107-server.virtnet: executing unload_modules_local [14728.622957] Key type lgssc unregistered [14728.976036] LNet: 392569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14728.981670] LNetError: 392569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14728.994203] LNet: Removed LNI 192.168.201.107@tcp [14730.136895] Key type .llcrypt unregistered [14730.139937] Key type ._llcrypt unregistered