[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 425949370 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002304] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006802] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.009113] LSM: Security Framework initializing [ 0.010034] Yama: becoming mindful. [ 0.011026] SELinux: Initializing. [ 0.013009] *** VALIDATE selinux *** [ 0.021609] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026356] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028073] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029090] *** VALIDATE tmpfs *** [ 0.031121] *** VALIDATE proc *** [ 0.032197] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.035170] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038023] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.042456] debug: unmapping init [mem 0xffffffff91859000-0xffffffff91860fff] [ 0.045155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046567] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047013] ... version: 2 [ 0.048007] ... bit width: 48 [ 0.049007] ... generic registers: 4 [ 0.050007] ... value mask: 0000ffffffffffff [ 0.051007] ... max period: 00007fffffffffff [ 0.052007] ... fixed-purpose events: 3 [ 0.053007] ... event mask: 000000070000000f [ 0.055198] rcu: Hierarchical SRCU implementation. [ 0.057493] smp: Bringing up secondary CPUs ... [ 0.058485] x86: Booting SMP configuration: [ 0.059012] .... node #0, CPUs: #1 #2 #3 [ 0.062020] smp: Brought up 1 node, 4 CPUs [ 0.064009] smpboot: Max logical packages: 1 [ 0.065012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234781] node 0 deferred pages initialised in 168ms [ 0.237150] devtmpfs: initialized [ 0.238211] x86/mm: Memory block size: 128MB [ 0.240000] gcov: version magic: 0x41383552 [ 0.242388] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244214] pinctrl core: initialized pinctrl subsystem [ 0.245000] [ 0.245000] ************************************************************* [ 0.247024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249007] ** ** [ 0.250005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251006] ** ** [ 0.253006] ** This means that this kernel is built to expose internal ** [ 0.254005] ** IOMMU data structures, which may compromise security on ** [ 0.255005] ** your system. ** [ 0.257006] ** ** [ 0.258005] ** If you see this message and you are not debugging the ** [ 0.259005] ** kernel, report this immediately to your vendor! ** [ 0.261007] ** ** [ 0.262005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263005] ************************************************************* [ 0.264648] NET: Registered protocol family 16 [ 0.266437] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.268028] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.270028] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.272389] cpuidle: using governor menu [ 0.273441] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.275343] PCI: Using configuration type 1 for base access [ 0.276088] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282151] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.284019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.287027] cryptd: max_cpu_qlen set to 1000 [ 0.289290] ACPI: Added _OSI(Module Device) [ 0.291012] ACPI: Added _OSI(Processor Device) [ 0.293010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.294006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299042] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.306272] ACPI: Interpreter enabled [ 0.308052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.309009] ACPI: Using IOAPIC for interrupt routing [ 0.312077] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.315257] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.324205] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.327008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.329035] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.332969] acpiphp: Slot [2] registered [ 0.334076] acpiphp: Slot [5] registered [ 0.335137] acpiphp: Slot [6] registered [ 0.336018] acpiphp: Slot [7] registered [ 0.336844] acpiphp: Slot [8] registered [ 0.338061] acpiphp: Slot [9] registered [ 0.338900] acpiphp: Slot [10] registered [ 0.340062] acpiphp: Slot [3] registered [ 0.340882] acpiphp: Slot [4] registered [ 0.341042] acpiphp: Slot [11] registered [ 0.341936] acpiphp: Slot [12] registered [ 0.343046] acpiphp: Slot [13] registered [ 0.343966] acpiphp: Slot [14] registered [ 0.345054] acpiphp: Slot [15] registered [ 0.345934] acpiphp: Slot [16] registered [ 0.346054] acpiphp: Slot [17] registered [ 0.346967] acpiphp: Slot [18] registered [ 0.348076] acpiphp: Slot [19] registered [ 0.350072] acpiphp: Slot [20] registered [ 0.352071] acpiphp: Slot [21] registered [ 0.354075] acpiphp: Slot [22] registered [ 0.356086] acpiphp: Slot [23] registered [ 0.357054] acpiphp: Slot [24] registered [ 0.359063] acpiphp: Slot [25] registered [ 0.361092] acpiphp: Slot [26] registered [ 0.362077] acpiphp: Slot [27] registered [ 0.364075] acpiphp: Slot [28] registered [ 0.366082] acpiphp: Slot [29] registered [ 0.368057] acpiphp: Slot [30] registered [ 0.369081] acpiphp: Slot [31] registered [ 0.371058] PCI host bridge to bus 0000:00 [ 0.373015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.376012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.378011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.381010] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.383009] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.384010] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.385137] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.388204] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.392372] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.403006] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.407755] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.410011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.412008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.414008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.416649] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.418890] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.424895] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.430012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.444019] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.449017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.455284] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.460008] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.464011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.476013] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.482101] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.486008] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.491021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502014] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.509426] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.513009] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.519011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.529012] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.536760] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.547011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.557011] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.564683] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.570009] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.576010] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.586011] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.594528] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.598008] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.602008] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.611013] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.616587] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.618203] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.619887] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.621192] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.623104] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.626233] iommu: Default domain type: Passthrough [ 0.627289] SCSI subsystem initialized [ 0.628077] ACPI: bus type USB registered [ 0.628965] usbcore: registered new interface driver usbfs [ 0.630029] usbcore: registered new interface driver hub [ 0.632030] usbcore: registered new device driver usb [ 0.633081] pps_core: LinuxPPS API ver. 1 registered [ 0.634006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.635015] PTP clock support registered [ 0.636089] EDAC MC: Ver: 3.0.0 [ 0.638085] PCI: Using ACPI for IRQ routing [ 0.639630] NetLabel: Initializing [ 0.640008] NetLabel: domain hash size = 128 [ 0.640828] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.642053] NetLabel: unlabeled traffic allowed by default [ 0.643098] vgaarb: loaded [ 0.644180] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.645006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.650005] clocksource: Switched to clocksource kvm-clock [ 0.747295] VFS: Disk quotas dquot_6.6.0 [ 0.748863] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.751359] *** VALIDATE ramfs *** [ 0.752585] *** VALIDATE hugetlbfs *** [ 0.754093] pnp: PnP ACPI init [ 0.756448] pnp: PnP ACPI: found 6 devices [ 0.770275] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.773518] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.775662] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.777849] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.780205] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.782596] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.785365] NET: Registered protocol family 2 [ 0.787840] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.792447] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.795860] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.801282] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.804906] TCP: Hash tables configured (established 65536 bind 65536) [ 0.807708] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.810509] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.813283] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.816053] NET: Registered protocol family 1 [ 0.818216] RPC: Registered named UNIX socket transport module. [ 0.820329] RPC: Registered udp transport module. [ 0.821854] RPC: Registered tcp transport module. [ 0.823390] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.825598] NET: Registered protocol family 44 [ 0.827159] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.829160] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.831073] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.833341] PCI: CLS 0 bytes, default 64 [ 0.834969] Unpacking initramfs... [ 2.363879] debug: unmapping init [mem 0xffff8b1e7cc54000-0xffff8b1e7ffbffff] [ 2.369507] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.371279] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.374576] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.854921] Initialise system trusted keyrings [ 2.856341] Key type blacklist registered [ 2.861053] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.875587] zbud: loaded [ 2.879188] *** VALIDATE nfs *** [ 2.880240] *** VALIDATE nfs4 *** [ 2.881840] pstore: using deflate compression [ 2.885202] Platform Keyring initialized [ 2.994053] NET: Registered protocol family 38 [ 2.995586] Key type asymmetric registered [ 2.996984] Asymmetric key parser 'x509' registered [ 2.998648] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.001977] io scheduler mq-deadline registered [ 3.003303] io scheduler kyber registered [ 3.004469] io scheduler bfq registered [ 3.006270] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.008360] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.009991] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.011929] ACPI: Power Button [PWRF] [ 3.112964] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.215545] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.405063] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.498409] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.693881] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.724938] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.754593] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.764088] Non-volatile memory driver v1.3 [ 3.765755] Linux agpgart interface v0.103 [ 3.812766] virtio_blk virtio1: [vda] 133040 512-byte logical blocks (68.1 MB/65.0 MiB) [ 3.815172] vda: detected capacity change from 0 to 68116480 [ 3.832311] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.835451] vdb: detected capacity change from 0 to 1073741824 [ 3.853759] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.856813] vdc: detected capacity change from 0 to 2621440000 [ 3.874208] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.877277] vdd: detected capacity change from 0 to 2621440000 [ 3.891463] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.894505] vde: detected capacity change from 0 to 4294967296 [ 3.909846] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.912086] vdf: detected capacity change from 0 to 4294967296 [ 3.918850] libphy: Fixed MDIO Bus: probed [ 3.925216] usbcore: registered new interface driver usbserial_generic [ 3.927447] usbserial: USB Serial support registered for generic [ 3.929563] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.933424] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.935230] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.938547] mousedev: PS/2 mouse device common for all mice [ 3.942173] rtc_cmos 00:05: RTC can wake from S4 [ 3.944681] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.948450] rtc_cmos 00:05: registered as rtc0 [ 3.950471] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.952186] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.953644] intel_pstate: CPU model not supported [ 3.958229] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.962793] hid: raw HID events driver (C) Jiri Kosina [ 3.964586] usbcore: registered new interface driver usbhid [ 3.966380] usbhid: USB HID core driver [ 3.967439] drop_monitor: Initializing network drop monitor service [ 3.969516] Initializing XFRM netlink socket [ 3.971534] NET: Registered protocol family 10 [ 3.975218] Segment Routing with IPv6 [ 3.976561] NET: Registered protocol family 17 [ 3.978786] mpls_gso: MPLS GSO support [ 3.984752] RAS: Correctable Errors collector initialized. [ 3.986633] AVX version of gcm_enc/dec engaged. [ 3.988236] AES CTR mode by8 optimization enabled [ 4.098659] sched_clock: Marking stable (4098375303, 0)->(4791969709, -693594406) [ 4.102304] registered taskstats version 1 [ 4.104565] Loading compiled-in X.509 certificates [ 4.107458] zswap: loaded using pool lzo/zbud [ 4.132696] Key type big_key registered [ 4.148293] Key type encrypted registered [ 4.149627] ima: No TPM chip found, activating TPM-bypass! [ 4.151194] ima: Allocated hash algorithm: sha1 [ 4.152711] ima: No architecture policies found [ 4.154411] evm: Initialising EVM extended attributes: [ 4.156175] evm: security.selinux [ 4.157297] evm: security.ima [ 4.158127] evm: security.capability [ 4.159376] evm: HMAC attrs: 0x1 [ 4.162280] rtc_cmos 00:05: setting system clock to 2025-08-31 21:30:08 UTC (1756675808) [ 4.171103] debug: unmapping init [mem 0xffffffff92803000-0xffffffff929fffff] [ 4.174323] debug: unmapping init [mem 0xffffffff91582000-0xffffffff91858fff] [ 4.181160] Write protecting the kernel read-only data: 28672k [ 4.185343] debug: unmapping init [mem 0xffffffff8fc03000-0xffffffff8fdfffff] [ 4.189306] debug: unmapping init [mem 0xffffffff90514000-0xffffffff905fffff] [ 4.244175] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.252773] systemd[1]: Detected virtualization kvm. [ 4.254873] systemd[1]: Detected architecture x86-64. [ 4.256609] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.283621] systemd[1]: No hostname configured. [ 4.285545] systemd[1]: Set hostname to . [ 4.287418] random: systemd: uninitialized urandom read (16 bytes read) [ 4.289624] systemd[1]: Initializing machine ID from random generator. [ 4.343152] random: ln: uninitialized urandom read (6 bytes read) [ 4.467804] random: systemd: uninitialized urandom read (16 bytes read) [ 4.471669] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.477572] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.485971] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.329910] device-mapper: uevent: version 1.0.3 [ 5.331915] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.253320] virtio_net virtio0 ens2: renamed from eth0 [ 6.286260] random: fast init done [ 6.474675] scsi host0: ata_piix [ 6.534275] scsi host1: ata_piix [ 6.535639] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.538016] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.681980] random: crng init done [ 10.684375] random: 7 urandom warning(s) missed due to ratelimiting [ 11.432406] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.733479] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.920070] printk: systemd: 21 output lines suppressed due to ratelimiting [ 16.374866] SELinux: Disabled at runtime. [ 16.465364] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.484161] systemd[1]: Detected virtualization kvm. [ 16.485941] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.327292] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.331243] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.338086] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.343258] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.348447] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.357915] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.374546] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 17.444303] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.091612] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.686660] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.794672] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.015231] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.032894] EDAC sbridge: Ver: 1.1.2 [ 21.085400] Key type dns_resolver registered [ 21.466947] NFS: Registering the id_resolver key type [ 21.468975] Key type id_resolver registered [ 21.470427] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg144-server login: [ 50.341790] libcfs: loading out-of-tree module taints kernel. [ 50.351844] Key type ._llcrypt registered [ 50.353219] Key type .llcrypt registered [ 50.392534] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 57.497294] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 58.114409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 58.120119] alg: No test for adler32 (adler32-zlib) [ 59.106114] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 59.372793] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 60.991105] Key type lgssc registered [ 61.482624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.900280] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 69.064143] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.581451] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 75.089215] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.683929] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.549019] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 86.879799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 86.900346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 86.909387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 88.021456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 88.034709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 88.078744] Lustre: lustre-MDT0000: new disk, initializing [ 88.103890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 88.109338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 89.379825] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.707712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.745599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 94.777608] Lustre: 6486:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 94.794170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 94.797096] Lustre: Skipped 1 previous similar message [ 94.843624] Lustre: lustre-MDT0001: new disk, initializing [ 94.867957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 94.877612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 94.881696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 96.293401] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.635533] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 102.027242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.055222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 102.159031] Lustre: lustre-OST0000: new disk, initializing [ 102.160573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 102.179493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 104.108437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 104.112757] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 104.135630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 104.184655] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.776660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.801607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 109.838982] Lustre: lustre-OST0001: new disk, initializing [ 109.840751] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 109.858954] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.778249] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 112.749644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 112.753457] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 112.769284] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 117.670367] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 120.276102] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 127.077835] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing check_logdir /tmp/testlogs/ [ 128.607923] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing yml_node [ 130.178715] Lustre: DEBUG MARKER: Client: 2.16.58.1 [ 131.025572] Lustre: DEBUG MARKER: MDS: 2.16.58.1 [ 131.852954] Lustre: DEBUG MARKER: OSS: 2.16.58.1 [ 132.386244] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 31 17:32:16 EDT 2025 [ 138.337971] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 138.820455] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 151.519545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 151.522063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 151.525863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 154.080241] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 154.080244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 154.084331] Lustre: Skipped 2 previous similar messages [ 156.541840] LustreError: 12190:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 156.577391] Lustre: server umount lustre-MDT0000 complete [ 157.800055] LustreError: 7420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756675962 with bad export cookie 17608606381726050968 [ 157.801157] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 157.803450] LustreError: 7420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 159.199787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 159.200072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 159.203814] Lustre: Skipped 1 previous similar message [ 159.205981] Lustre: Skipped 3 previous similar messages [ 162.210172] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 162.212593] Lustre: Skipped 1 previous similar message [ 163.899422] LustreError: 12392:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 163.901107] LustreError: 12392:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 163.986790] Lustre: server umount lustre-MDT0001 complete [ 165.282091] LustreError: 12593:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 165.284284] LustreError: 12593:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 165.303262] Lustre: server umount lustre-OST0000 complete [ 166.621463] Lustre: server umount lustre-OST0001 complete [ 168.593738] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 171.370377] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 174.572339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.738449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.675948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 180.737664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 183.027574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 183.051448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.151585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 183.164487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.202223] Lustre: lustre-MDT0000: new disk, initializing [ 183.229091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.236989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.603091] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 188.739453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 188.761783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 188.789983] Lustre: 14985:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.899997] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.158743] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 193.023864] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 194.197675] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 196.574046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.595972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 196.705557] Lustre: lustre-OST0000: new disk, initializing [ 196.707206] Lustre: Skipped 1 previous similar message [ 196.708965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.711691] Lustre: Skipped 2 previous similar messages [ 196.743703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.745573] Lustre: Skipped 1 previous similar message [ 198.208249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.210956] Lustre: Skipped 1 previous similar message [ 198.212711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.251574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.683168] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 201.829823] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 203.226562] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 203.320747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 204.670403] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 204.754372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 208.354953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.356628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 208.359098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 208.362803] Lustre: Skipped 1 previous similar message [ 211.438186] LustreError: 17919:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 211.440883] LustreError: 17919:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 211.514366] Lustre: server umount lustre-OST0000 complete [ 213.473527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 213.477037] Lustre: Skipped 2 previous similar messages [ 218.592626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 218.595295] Lustre: Skipped 3 previous similar messages [ 227.295103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 227.328310] LustreError: 18120:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 227.331898] LustreError: 18120:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 227.364767] Lustre: server umount lustre-MDT0000 complete [ 227.808769] LustreError: 15771:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 227.814536] LustreError: 15771:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 6 previous similar messages [ 228.671456] LustreError: 14976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676033 with bad export cookie 17608606381726052494 [ 228.674119] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 228.674376] LustreError: 14976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 228.830358] Lustre: server umount lustre-MDT0001 complete [ 230.410533] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 17:33:54 (1756676034) [ 232.654342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 232.819344] LustreError: 18848:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.846617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.252513] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 237.283173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 238.865511] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 240.832833] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 242.057600] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 244.381093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 246.450158] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 248.168160] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 265.420774] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 269.283103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.284453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.286644] Lustre: Skipped 1 previous similar message [ 269.288373] Lustre: Skipped 2 previous similar messages [ 274.559675] LustreError: 21173:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 274.562315] LustreError: 21173:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 274.596116] Lustre: server umount lustre-MDT0000 complete [ 275.843454] LustreError: 21069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676080 with bad export cookie 17608606381726053376 [ 275.849861] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 276.009798] Lustre: server umount lustre-MDT0001 complete [ 287.546366] Lustre: server umount lustre-OST0000 complete [ 294.164826] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 297.731993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 297.882067] LustreError: 22899:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 297.900630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 297.902643] Lustre: Skipped 2 previous similar messages [ 299.185039] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 302.079479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 303.651572] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 304.642707] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 306.985389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 308.988975] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 311.937721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 311.959462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 312.015874] Lustre: lustre-OST0001: new disk, initializing [ 312.018467] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 313.860434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 313.863643] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 313.910524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 314.089912] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 318.595655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 326.165777] Lustre: Setting parameter general.obdfilter.*.client_cache_count=256 in log params [ 326.168258] Lustre: Skipped 1 previous similar message [ 329.184889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.188210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.191172] Lustre: Skipped 4 previous similar messages [ 329.196303] Lustre: Skipped 8 previous similar messages [ 335.230205] LustreError: 26378:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 335.231845] LustreError: 26378:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 335.261953] Lustre: server umount lustre-MDT0000 complete [ 336.506393] LustreError: 22878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676140 with bad export cookie 17608606381726054223 [ 336.509347] LustreError: 22878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 336.510509] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 336.669447] Lustre: server umount lustre-MDT0001 complete [ 348.673791] Lustre: server umount lustre-OST0000 complete [ 360.248339] Lustre: server umount lustre-OST0001 complete [ 365.615594] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 369.076372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 369.217205] LustreError: 28158:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 369.240794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.244104] Lustre: Skipped 3 previous similar messages [ 370.548163] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 373.363235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 374.849662] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 375.855407] Lustre: 29273:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 378.168709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 379.300355] LustreError: 29629:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 380.223587] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 383.099958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 385.126948] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 388.200219] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 389.335540] Lustre: 31130:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 399.327874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 399.329943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.333569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 399.336955] Lustre: Skipped 4 previous similar messages [ 402.044392] LustreError: 31333:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 402.046409] LustreError: 31333:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 402.080371] Lustre: server umount lustre-MDT0000 complete [ 403.364555] LustreError: 31247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676207 with bad export cookie 17608606381726055518 [ 403.370137] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 403.520237] Lustre: server umount lustre-MDT0001 complete [ 415.168525] Lustre: server umount lustre-OST0000 complete [ 426.736496] Lustre: server umount lustre-OST0001 complete [ 429.544811] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 17:37:13 (1756676233) [ 438.459357] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 441.784882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.928365] LustreError: 33912:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 443.203615] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 446.038509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 447.519575] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 448.458581] Lustre: 35027:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 450.678897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 452.678424] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 455.584831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 457.750462] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 460.981068] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 462.136966] Lustre: 36880:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 472.033535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 472.034753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.039969] Lustre: Skipped 6 previous similar messages [ 474.560574] LustreError: 37033:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 474.563713] LustreError: 37033:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 474.595767] Lustre: server umount lustre-MDT0000 complete [ 475.946236] LustreError: 33892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676280 with bad export cookie 17608606381726056624 [ 475.949105] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.949799] LustreError: 33892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 476.116808] Lustre: server umount lustre-MDT0001 complete [ 487.682116] Lustre: server umount lustre-OST0000 complete [ 499.260629] Lustre: server umount lustre-OST0001 complete [ 502.105619] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 17:38:25 (1756676305) [ 506.775873] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 510.262428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.404536] LustreError: 39004:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.413076] LustreError: 39004:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 510.425901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.429115] Lustre: Skipped 7 previous similar messages [ 511.694526] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 514.381628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 515.817578] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 516.754799] Lustre: 40120:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 518.881948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 520.868757] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 523.786901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 525.730505] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 528.723389] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 535.133621] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 537.009809] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.44@tcp (stopping) [ 537.011699] Lustre: Skipped 7 previous similar messages [ 540.128296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 542.361704] Lustre: server umount lustre-MDT0000 complete [ 543.609099] LustreError: 41979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676347 with bad export cookie 17608606381726057730 [ 543.612832] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.614085] LustreError: 41979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 545.760123] LustreError: 40503:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.764721] LustreError: 40503:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 549.832975] Lustre: server umount lustre-MDT0001 complete [ 552.713974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.863719] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b1dc2f72300 x1842008150023040/t0(0) o250->MGC192.168.201.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 554.379303] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 557.383748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 559.015124] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 560.750470] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 572.908043] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 577.819158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec [ 578.023730] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 578.025741] Lustre: Skipped 1 previous similar message [ 579.013925] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 579.550150] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 583.142333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 586.971715] Lustre: server umount lustre-MDT0000 complete [ 588.215272] LustreError: 41979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676392 with bad export cookie 17608606381726058871 [ 588.218460] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 588.218932] LustreError: 41979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 588.362075] Lustre: server umount lustre-MDT0001 complete [ 595.646383] Lustre: server umount lustre-OST0000 complete [ 596.976830] Lustre: server umount lustre-OST0001 complete [ 599.814982] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 17:40:03 (1756676403) [ 604.946584] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 608.558783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.100806] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 613.042259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 614.553292] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 615.521777] Lustre: 47476:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 615.524155] Lustre: 47476:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message [ 617.743706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 618.913988] LustreError: 47832:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.918735] LustreError: 47832:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 9 previous similar messages [ 619.863866] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 622.622732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 624.641379] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 627.716278] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 634.493925] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 638.944066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 638.947304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.951762] Lustre: Skipped 12 previous similar messages [ 644.928389] LustreError: 49547:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 644.930758] LustreError: 49547:0:(obd_class.h:479:obd_check_dev()) Skipped 67 previous similar messages [ 644.964980] Lustre: server umount lustre-MDT0000 complete [ 646.409963] Lustre: server umount lustre-MDT0001 complete [ 657.919152] Lustre: server umount lustre-OST0000 complete [ 669.489386] Lustre: server umount lustre-OST0001 complete [ 672.531093] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 17:41:16 (1756676476) [ 673.350900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 673.902937] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 17:41:17 (1756676477) [ 677.336630] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 680.036274] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 683.313737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 685.224608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 687.442198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 687.463176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 687.557085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 687.596392] Lustre: lustre-MDT0000: new disk, initializing [ 687.626666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 688.952514] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 691.910943] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 694.137881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 694.158132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 695.936826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 695.945312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 696.080198] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 699.118392] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 699.782242] Lustre: DEBUG MARKER: create test files [ 711.648896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 715.099724] Lustre: server umount lustre-OST0000 complete [ 716.434257] Lustre: server umount lustre-MDT0000 complete [ 719.992500] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 721.075821] Key type lgssc unregistered [ 721.215713] LNet: 54863:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.217961] LNetError: 54863:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.224314] LNet: Removed LNI 192.168.201.144@tcp [ 721.576113] Key type .llcrypt unregistered [ 721.577282] Key type ._llcrypt unregistered [ 731.663061] Key type ._llcrypt registered [ 731.664024] Key type .llcrypt registered [ 731.703818] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 731.967868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 731.995153] alg: No test for adler32 (adler32-zlib) [ 732.844911] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 732.924380] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 734.503125] Key type lgssc registered [ 734.867613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 737.474424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 737.485311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 738.640591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.956474] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 741.665241] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.839257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 743.939587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 745.729771] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 747.435488] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 748.070889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 748.099835] Lustre: 57445:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 748.102469] Lustre: 57445:0:(osd_handler.c:2079:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 748.104371] Lustre: 57445:0:(osd_handler.c:2086:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 748.106010] Lustre: 57445:0:(osd_handler.c:2096:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 748.108394] Lustre: 57445:0:(osd_handler.c:2103:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 748.111564] Lustre: 57445:0:(osd_handler.c:2110:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.874289] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 757.435648] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 763.360802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 763.361030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.367223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 765.357207] LustreError: 58189:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 765.434775] Lustre: server umount lustre-OST0000 complete [ 766.731402] LustreError: 58391:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 766.733846] LustreError: 58391:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 766.818347] Lustre: server umount lustre-MDT0000 complete [ 770.289329] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 771.453118] Key type lgssc unregistered [ 771.590514] LNet: 58923:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.592720] LNetError: 58923:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.599477] LNet: Removed LNI 192.168.201.144@tcp [ 771.940679] Key type .llcrypt unregistered [ 771.941744] Key type ._llcrypt unregistered [ 783.738695] Key type ._llcrypt registered [ 783.739792] Key type .llcrypt registered [ 783.783099] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 784.215402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 784.224717] alg: No test for adler32 (adler32-zlib) [ 785.101453] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 785.203686] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 786.791187] Key type lgssc registered [ 787.203489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 789.915296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 789.922537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.113645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.418431] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 794.181649] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 796.411787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 796.519532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.390969] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 800.111602] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 801.636383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 801.657468] Lustre: 61604:0:(osd_internal.h:1341:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 801.661474] Lustre: 61604:0:(osd_handler.c:2079:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 801.664578] Lustre: 61604:0:(osd_handler.c:2086:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 801.667691] Lustre: 61604:0:(osd_handler.c:2096:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 801.669615] Lustre: 61604:0:(osd_handler.c:2103:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 801.671975] Lustre: 61604:0:(osd_handler.c:2110:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.091585] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 812.000797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 812.000892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.007550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 812.206062] LustreError: 62296:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 812.259319] Lustre: server umount lustre-OST0000 complete [ 815.489154] LustreError: 62496:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 815.491314] LustreError: 62496:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 815.568570] Lustre: server umount lustre-MDT0000 complete [ 818.929598] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 819.916642] Key type lgssc unregistered [ 820.047418] LNet: 63029:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.052173] LNetError: 63029:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.062346] LNet: Removed LNI 192.168.201.144@tcp [ 820.370214] Key type .llcrypt unregistered [ 820.371449] Key type ._llcrypt unregistered [ 826.042893] Key type ._llcrypt registered [ 826.044247] Key type .llcrypt registered [ 826.085714] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 831.958551] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 832.281719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 832.315068] alg: No test for adler32 (adler32-zlib) [ 833.155398] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 833.237257] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 834.815245] Key type lgssc registered [ 835.151399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 837.914237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 840.016797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 841.952461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 843.987398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 846.170355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 846.191314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 846.196157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.277722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 847.291822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 847.325856] Lustre: lustre-MDT0000: new disk, initializing [ 847.353702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.360942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 848.632221] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 852.671723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 852.694897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 852.725363] Lustre: 66142:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 852.749263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 852.750724] Lustre: Skipped 1 previous similar message [ 852.783707] Lustre: lustre-MDT0001: new disk, initializing [ 852.815162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 852.827863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 852.831132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 854.107533] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 857.282484] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 858.491644] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 860.740357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 860.765414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 860.868937] Lustre: lustre-OST0000: new disk, initializing [ 860.870983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 860.905736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.765536] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 863.416792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 863.419957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 863.456199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 865.861729] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 867.272406] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 867.365336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 868.699913] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 868.778571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 871.470085] LustreError: 69079:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 871.526743] Lustre: server umount lustre-OST0000 complete [ 873.439973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 873.444583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.452524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.952639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.956559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.072451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.074763] Lustre: Skipped 1 previous similar message [ 884.192139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.193711] Lustre: Skipped 1 previous similar message [ 887.263156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 887.300684] LustreError: 69280:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 887.302338] LustreError: 69280:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 887.329843] Lustre: server umount lustre-MDT0000 complete [ 888.520139] LustreError: 66135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676692 with bad export cookie 4159685314036233403 [ 888.523445] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 888.523994] LustreError: 66135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 898.019767] LustreError: 69483:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 898.022476] LustreError: 69483:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 898.102464] Lustre: server umount lustre-MDT0001 complete [ 900.925033] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 17:45:04 (1756676704) [ 904.417074] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 908.865946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 917.171118] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 920.209338] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 923.412353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 925.551465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 927.509676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 929.637584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 931.852235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 931.871216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.952691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 931.964252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 931.999868] Lustre: lustre-MDT0000: new disk, initializing [ 932.024457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.029985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 933.327151] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 937.334910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 937.357210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 937.388070] Lustre: 73715:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 937.425218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 937.428348] Lustre: Skipped 1 previous similar message [ 937.471338] Lustre: lustre-MDT0001: new disk, initializing [ 937.505988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.517245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 937.519979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 938.749481] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 941.740266] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 942.862854] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 945.055506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 945.080838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 945.845205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 945.850152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 945.889196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 947.084438] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 950.211654] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 951.571444] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 951.651855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 952.931503] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 953.010638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 956.386522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 956.386619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.389762] LustreError: Skipped 1 previous similar message [ 956.393260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.395052] Lustre: Skipped 1 previous similar message [ 959.724727] LustreError: 76653:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 959.728073] LustreError: 76653:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 959.789682] Lustre: server umount lustre-OST0000 complete [ 961.504424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 961.507505] Lustre: Skipped 2 previous similar messages [ 966.623920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.625344] Lustre: Skipped 3 previous similar messages [ 975.327149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 975.377467] LustreError: 76855:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 975.380607] LustreError: 76855:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 975.414182] Lustre: server umount lustre-MDT0000 complete [ 975.840176] LustreError: 73726:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.844768] LustreError: 73726:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 976.871677] LustreError: 74494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676781 with bad export cookie 4159685314036234271 [ 976.875430] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.877174] LustreError: 74494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 977.027611] Lustre: server umount lustre-MDT0001 complete [ 980.161611] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 17:46:23 (1756676783) [ 982.412986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 982.565437] LustreError: 77729:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 982.588360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.590094] Lustre: Skipped 1 previous similar message [ 983.837717] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 986.643170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 988.033511] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 990.063293] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 991.232151] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 993.386404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 995.337861] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 996.997606] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 998.179808] Lustre: 80066:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting 2ac8fc94-b804-4027-915d-5fcd841ce9ce at adminstrative request [ 1002.977751] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.982401] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1013.216588] Lustre: MGS: Client 2ac8fc94-b804-4027-915d-5fcd841ce9ce (at 0@lo) reconnecting [ 1013.219820] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x39ba2bc11089fb75 to 0x39ba2bc11089fe00 [ 1013.223615] Lustre: MGC192.168.201.144@tcp: Connection restored to 0@lo (at 0@lo) [ 1030.618459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1030.647418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1030.700338] Lustre: lustre-OST0001: new disk, initializing [ 1030.701762] Lustre: Skipped 1 previous similar message [ 1030.703411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1030.704871] Lustre: Skipped 1 previous similar message [ 1030.730772] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1030.732396] Lustre: Skipped 2 previous similar messages [ 1032.576280] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1035.648188] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1036.578236] LustreError: 81193:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1036.581701] LustreError: 81193:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1036.598049] Lustre: server umount lustre-OST0001 complete [ 1037.302969] LustreError: 79327:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1037.791637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1037.793534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.796632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1037.798645] Lustre: Skipped 2 previous similar messages [ 1042.403376] LustreError: 79325:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1042.410430] LustreError: 79325:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1047.523513] LustreError: 80080:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.528838] LustreError: 80080:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1052.127167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1052.207470] Lustre: server umount lustre-OST0000 complete [ 1053.663872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1053.667118] LustreError: Skipped 1 previous similar message [ 1053.668886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.673747] Lustre: Skipped 1 previous similar message [ 1059.579736] LustreError: 81606:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 1059.581942] LustreError: 81606:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1059.607382] Lustre: server umount lustre-MDT0000 complete [ 1060.830206] LustreError: 77710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676865 with bad export cookie 4159685314036235776 [ 1060.833044] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1060.835496] LustreError: 77710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1060.963853] Lustre: server umount lustre-MDT0001 complete [ 1063.678181] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 17:47:47 (1756676867) [ 1064.169538] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1064.731241] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 17:47:48 (1756676868) [ 1065.271882] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1065.882500] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 17:47:49 (1756676869) [ 1066.484795] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1067.097934] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 17:47:50 (1756676870) [ 1067.676580] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1069.298860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1069.644038] loop: module loaded [ 1069.648441] loop0: detected capacity change from 0 to 4294967296 [ 1069.661221] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1070.145466] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1071.060666] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1072.404978] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1075.458780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1076.774040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1078.042649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1079.303268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1081.490156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1081.512361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1081.522996] LustreError: 84377:0:(obd_config.c:782:class_setup()) setup lustre-OST0000-osd failed (-28) [ 1081.525346] LustreError: 84377:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 1081.527289] LustreError: 84377:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 1081.529475] LustreError: 84377:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 1087.636333] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 1090.719301] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 1094.203698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1096.423196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1098.441209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1100.495200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1102.652222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1102.673754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.752619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1102.763838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1102.803062] Lustre: lustre-MDT0000: new disk, initializing [ 1102.825894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.835336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1104.165457] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1108.340308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1108.368725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1108.398233] Lustre: 87212:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1108.508515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1109.743738] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1112.728787] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1113.889605] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1116.072199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1116.096554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1118.044405] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1119.742710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1119.785751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1121.105207] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1122.422195] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1122.492461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1123.817291] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1123.892924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1124.834080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1124.834390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.836097] LustreError: Skipped 1 previous similar message [ 1124.838750] Lustre: Skipped 1 previous similar message [ 1124.839391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.842650] Lustre: Skipped 9 previous similar messages [ 1126.574104] LustreError: 90150:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1126.575819] LustreError: 90150:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1126.638243] Lustre: server umount lustre-OST0000 complete [ 1142.239137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1142.293905] Lustre: server umount lustre-MDT0000 complete [ 1143.512131] LustreError: 87205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676947 with bad export cookie 4159685314036236364 [ 1143.515023] LustreError: 87205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1143.515985] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.661750] Lustre: server umount lustre-MDT0001 complete [ 1146.441772] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 17:49:10 (1756676950) [ 1148.633917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.784844] LustreError: 91223:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.150941] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1152.939307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1154.319070] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1156.157438] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1157.310096] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1159.540814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1161.553736] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1163.308985] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1164.674534] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1174.966853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 1176.329704] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1176.416837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1178.619792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1178.643455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1178.697871] Lustre: lustre-OST0001: new disk, initializing [ 1178.699891] Lustre: Skipped 2 previous similar messages [ 1178.701946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1178.703857] Lustre: Skipped 3 previous similar messages [ 1178.732330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1178.736019] Lustre: Skipped 5 previous similar messages [ 1180.564958] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1183.603261] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1184.943454] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1187.841264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1187.844249] Lustre: Skipped 2 previous similar messages [ 1187.846309] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1187.877394] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1188.091050] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1189.398189] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1189.473987] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1199.222559] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1204.317926] Lustre: Failing over lustre-MDT0000 [ 1204.722492] LustreError: 96197:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1204.725453] LustreError: 96197:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 1204.762323] Lustre: server umount lustre-MDT0000 complete [ 1206.195133] LustreError: 92579:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.203991] LustreError: 92579:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1208.289520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.293388] Lustre: Skipped 5 previous similar messages [ 1219.459048] LDISKFS-fs (dm-0): recovery complete [ 1219.460953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.501669] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1219.624101] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1220.880854] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1221.548188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 1221.549028] Lustre: 95133:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1224.673065] Lustre: 96313:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 1224.677037] Lustre: 96313:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 1 previous similar message [ 1224.680912] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1224.697957] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1225.015111] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1225.329130] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1225.331595] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 1 previous similar message [ 1225.647078] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1225.649008] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 1 previous similar message [ 1225.969445] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1225.974257] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 8 previous similar messages [ 1226.600887] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1226.603744] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 3 previous similar messages [ 1226.919126] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1226.921753] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 3 previous similar messages [ 1228.175831] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1228.178869] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 6 previous similar messages [ 1228.807817] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1228.811901] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 6 previous similar messages [ 1229.127109] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1229.129138] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 6 previous similar messages [ 1232.257211] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1232.260977] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 12 previous similar messages [ 1232.889688] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1232.894095] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 12 previous similar messages [ 1233.207153] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1233.209500] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 12 previous similar messages [ 1240.393652] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1240.396914] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 25 previous similar messages [ 1241.024956] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1241.028607] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 25 previous similar messages [ 1241.343056] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1241.345417] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 25 previous similar messages [ 1256.649028] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 1256.653813] Lustre: 96943:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 51 previous similar messages [ 1257.280945] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 1257.285025] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 51 previous similar messages [ 1257.599077] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 1257.600971] LustreError: 96943:0:(ldlm_lib.c:2632:replay_request_or_update()) Skipped 51 previous similar messages [ 1281.007081] Lustre: 96943:0:(ldlm_lib.c:2328:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 1281.011459] Lustre: 96943:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1281.013746] Lustre: 96943:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client e6b02e31-b557-4d10-a29b-d3d3a32de8e9@192.168.201.44@tcp [ 1281.016757] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1281.018570] Lustre: 96943:0:(ldlm_lib.c:1850:abort_req_replay_queue()) @@@ aborted: req@000000008cf36e0e x1842008955095040/t0(8589934777) o101->e6b02e31-b557-4d10-a29b-d3d3a32de8e9@192.168.201.44@tcp:476/0 lens 592/0 e 0 to 0 dl 1756677096 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 1281.025795] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1281.031151] Lustre: lustre-MDT0000: Denying connection for new client e6b02e31-b557-4d10-a29b-d3d3a32de8e9 (at 192.168.201.44@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 1281.031889] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1281.046727] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1281.046870] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 1281.049467] Lustre: Skipped 2 previous similar messages [ 1281.068033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 1281.068043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 1290.719768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1290.721742] LustreError: Skipped 1 previous similar message [ 1290.722891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.725616] Lustre: Skipped 1 previous similar message [ 1290.727179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1290.729377] Lustre: Skipped 7 previous similar messages [ 1292.985653] Lustre: server umount lustre-OST0000 complete [ 1296.354152] LustreError: 93325:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.357886] LustreError: 93325:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 17 previous similar messages [ 1300.328688] Lustre: server umount lustre-OST0001 complete [ 1307.687330] Lustre: server umount lustre-MDT0000 complete [ 1308.966509] LustreError: 91248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677113 with bad export cookie 4159685314036287548 [ 1308.969706] LustreError: 91248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1308.971069] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1309.138283] Lustre: server umount lustre-MDT0001 complete [ 1311.889577] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 17:51:55 (1756677115) [ 1314.313802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1314.316069] Lustre: *** cfs_fail_loc=197, val=0*** [ 1331.807471] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1337.951391] LustreError: 98894:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.144@tcp: failed processing log, type 4: rc = -110 [ 1343.081203] Lustre: *** cfs_fail_loc=197, val=0*** [ 1343.083306] Lustre: Skipped 65 previous similar messages [ 1366.623360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.626696] Lustre: Skipped 1 previous similar message [ 1368.485433] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1370.192486] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1370.847119] LustreError: 99559:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1370.849680] LustreError: 99559:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1370.911674] Lustre: server umount lustre-OST0000 complete [ 1373.489331] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 17:52:57 (1756677177) [ 1386.918113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.107787] LustreError: 100402:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.414045] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1391.380808] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1393.368907] LustreError: 100432:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1393.371744] LustreError: 100432:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1393.374038] LustreError: 100432:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1393.526453] Lustre: server umount lustre-MDT0000 complete [ 1398.639473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1404.002128] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 1407.005474] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 1410.415154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1412.671275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1415.249097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1417.557023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1419.965042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1419.990686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.081816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1420.093804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1420.130224] Lustre: lustre-MDT0000: new disk, initializing [ 1420.158124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1421.553302] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1425.782646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1425.805786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1425.834222] Lustre: 104466:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1425.930108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1427.129083] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1430.104202] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1431.214650] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1433.489274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1433.513738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1435.429147] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1437.276746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1438.500988] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1439.877211] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1439.954619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1441.256533] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1441.339352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1442.273697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1442.274096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.275545] LustreError: Skipped 3 previous similar messages [ 1442.278445] Lustre: Skipped 5 previous similar messages [ 1442.278976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1442.282621] Lustre: Skipped 8 previous similar messages [ 1444.068961] Lustre: server umount lustre-OST0000 complete [ 1451.351266] Lustre: server umount lustre-MDT0000 complete [ 1452.574737] LustreError: 105245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677256 with bad export cookie 4159685314036298174 [ 1452.579528] LustreError: 105245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1452.579654] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1452.723703] Lustre: server umount lustre-MDT0001 complete [ 1455.508135] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 17:54:19 (1756677259) [ 1458.397027] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 1459.506705] Key type lgssc unregistered [ 1459.649484] LNet: 108531:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.652637] LNetError: 108531:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.665424] LNet: Removed LNI 192.168.201.144@tcp [ 1459.971280] Key type .llcrypt unregistered [ 1459.974109] Key type ._llcrypt unregistered [ 1465.675732] Key type ._llcrypt registered [ 1465.677302] Key type .llcrypt registered [ 1465.725248] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 1471.844693] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 1472.118223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1472.131038] alg: No test for adler32 (adler32-zlib) [ 1472.978472] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1473.067487] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 1474.647213] Key type lgssc registered [ 1475.003822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.984397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1480.118928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1482.027093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1484.261572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1486.133587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1488.325601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1488.350900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1488.358220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.435367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1489.447713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1489.488608] Lustre: lustre-MDT0000: new disk, initializing [ 1489.511959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.518112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1490.783073] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1494.890107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1494.918972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1495.024323] Lustre: lustre-OST0000: new disk, initializing [ 1495.026029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1495.028292] Lustre: Skipped 1 previous similar message [ 1495.054918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.938225] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1498.496514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1498.500153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1498.510441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1501.125848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1501.153213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1501.203593] Lustre: lustre-OST0001: new disk, initializing [ 1501.205765] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1501.229547] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1503.137948] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1504.897532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 1504.899972] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 1504.907352] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 1510.368585] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.369190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1510.373609] Lustre: Skipped 1 previous similar message [ 1510.376117] Lustre: Skipped 1 previous similar message [ 1513.275328] LustreError: 114086:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1513.352936] Lustre: server umount lustre-MDT0000 complete [ 1515.443851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1516.898405] LustreError: 111839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677321 with bad export cookie 15449018022025563724 [ 1516.900219] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1516.902460] LustreError: 111839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1516.922014] LustreError: 114679:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1516.924612] LustreError: 114679:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1516.940145] Lustre: server umount lustre-OST0000 complete [ 1518.170102] LustreError: 114880:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1518.171810] LustreError: 114880:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1518.236687] Lustre: server umount lustre-OST0001 complete [ 1521.139054] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 17:55:24 (1756677324) [ 1524.074856] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 1526.926826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1527.259237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1529.015920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1529.352057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1531.413111] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 17:55:35 (1756677335) [ 1535.419633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1536.042668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1536.666046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1537.289622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1537.610264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1538.224759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1543.515691] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 1546.454397] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 1549.730251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1551.940108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1554.045462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1556.136543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1558.407682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1558.427281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1558.509038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1558.521627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1558.555734] Lustre: lustre-MDT0000: new disk, initializing [ 1558.581141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1558.589714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1559.930452] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1564.010442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1564.039156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1564.074305] Lustre: 120192:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1564.103722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1564.105954] Lustre: Skipped 1 previous similar message [ 1564.148854] Lustre: lustre-MDT0001: new disk, initializing [ 1564.182255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1564.194211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1564.198717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1565.619392] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1568.723490] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1569.955358] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1572.281567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1572.301846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1572.395644] Lustre: lustre-OST0000: new disk, initializing [ 1572.397326] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1572.424236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.398453] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1577.530897] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1578.921451] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1582.077771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1582.083146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1582.121995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1583.084846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 1584.520380] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1584.624633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1587.170961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.171254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1587.176080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.178287] LustreError: Skipped 1 previous similar message [ 1587.180174] Lustre: Skipped 1 previous similar message [ 1591.341733] LustreError: 123160:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1591.344161] LustreError: 123160:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1591.406791] Lustre: server umount lustre-OST0000 complete [ 1594.847805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1594.851031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.856520] Lustre: Skipped 1 previous similar message [ 1594.859298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1597.408082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1597.408680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1598.715849] LustreError: 123362:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1598.719961] LustreError: 123362:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1598.754380] Lustre: server umount lustre-MDT0000 complete [ 1600.003362] LustreError: 120182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677404 with bad export cookie 15449018022025564935 [ 1600.007840] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1600.008346] LustreError: 120182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1600.158330] Lustre: server umount lustre-MDT0001 complete [ 1602.945705] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 17:56:46 (1756677406) [ 1605.214271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.385424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.698696] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1609.536912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1610.997814] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1612.824110] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1614.037802] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1616.331675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1618.357569] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1620.141675] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1632.749836] Lustre: *** cfs_fail_loc=159, val=0*** [ 1648.045239] Lustre: lustre-MDT0000: Client bca0bc8e-8542-4942-8cc9-9424ed9325ad (at 192.168.201.44@tcp) reconnecting [ 1648.531228] Lustre: *** cfs_fail_loc=159, val=0*** [ 1648.532477] Lustre: Skipped 5 previous similar messages [ 1663.928140] Lustre: lustre-MDT0000: Client bca0bc8e-8542-4942-8cc9-9424ed9325ad (at 192.168.201.44@tcp) reconnecting [ 1681.379094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1681.382355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.383681] LustreError: Skipped 1 previous similar message [ 1681.384688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.384702] Lustre: Skipped 1 previous similar message [ 1681.389509] Lustre: Skipped 1 previous similar message [ 1685.868941] LustreError: 126744:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1685.871380] LustreError: 126744:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1685.943336] Lustre: server umount lustre-OST0000 complete [ 1691.617173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.617221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1691.617921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1691.617926] Lustre: Skipped 1 previous similar message [ 1691.620745] Lustre: Skipped 1 previous similar message [ 1693.394777] Lustre: server umount lustre-MDT0000 complete [ 1694.574429] LustreError: 125851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677498 with bad export cookie 15449018022025565803 [ 1694.577850] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1694.579281] LustreError: 125851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1694.728801] Lustre: server umount lustre-MDT0001 complete [ 1698.653541] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 1699.697357] Key type lgssc unregistered [ 1699.837423] LNet: 127679:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.840868] LNetError: 127679:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.850286] LNet: Removed LNI 192.168.201.144@tcp [ 1700.157823] Key type .llcrypt unregistered [ 1700.158796] Key type ._llcrypt unregistered [ 1709.457142] Key type ._llcrypt registered [ 1709.458093] Key type .llcrypt registered [ 1709.493389] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 1709.823253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1709.856070] alg: No test for adler32 (adler32-zlib) [ 1710.711514] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1710.795505] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 1712.383192] Key type lgssc registered [ 1712.750895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1715.248252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1715.254644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.411648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1717.668746] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1720.486457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1720.612854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1722.015956] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1723.858639] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1725.016749] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1727.203216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1727.314896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.206120] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1730.994757] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1741.671833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 1746.342896] Lustre: *** cfs_fail_loc=159, val=0*** [ 1762.741319] Lustre: lustre-MDT0000: Client 04f3a125-dc97-481d-9bfb-1692a858315b (at 192.168.201.44@tcp) reconnecting [ 1763.451724] Lustre: *** cfs_fail_loc=159, val=0*** [ 1779.638850] Lustre: lustre-MDT0001: Client 04f3a125-dc97-481d-9bfb-1692a858315b (at 192.168.201.44@tcp) reconnecting [ 1780.103713] Lustre: *** cfs_fail_loc=159, val=0*** [ 1780.105106] Lustre: Skipped 3 previous similar messages [ 1796.532338] Lustre: lustre-MDT0001: Client 04f3a125-dc97-481d-9bfb-1692a858315b (at 192.168.201.44@tcp) reconnecting [ 1797.675789] Lustre: 131776:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1798.458773] Lustre: *** cfs_fail_loc=159, val=0*** [ 1798.461065] Lustre: Skipped 4 previous similar messages [ 1813.950690] Lustre: lustre-MDT0000: Client 9b23603c-e02e-4aa4-a50e-c78d70388f1b (at 192.168.201.44@tcp) reconnecting [ 1814.544365] Lustre: *** cfs_fail_loc=159, val=0*** [ 1814.545620] Lustre: Skipped 14 previous similar messages [ 1830.828538] Lustre: lustre-MDT0000: Client 9b23603c-e02e-4aa4-a50e-c78d70388f1b (at 192.168.201.44@tcp) reconnecting [ 1831.393632] Lustre: 132068:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1832.927827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1832.930702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.934404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.952799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1833.953943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1833.954613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.962144] Lustre: Skipped 1 previous similar message [ 1838.702099] LustreError: 132166:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1838.758129] Lustre: server umount lustre-OST0000 complete [ 1843.679823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1843.681722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.685046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1846.202400] LustreError: 132367:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1846.203867] LustreError: 132367:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1846.227579] Lustre: server umount lustre-MDT0000 complete [ 1847.423243] LustreError: 129034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677651 with bad export cookie 14819170553722138190 [ 1847.427041] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1847.428678] LustreError: 129034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1847.491656] LustreError: 132569:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1847.493414] LustreError: 132569:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1847.570483] Lustre: server umount lustre-MDT0001 complete [ 1851.220590] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 1852.273576] Key type lgssc unregistered [ 1852.409385] LNet: 133100:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.411748] LNetError: 133100:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.421299] LNet: Removed LNI 192.168.201.144@tcp [ 1852.715052] Key type .llcrypt unregistered [ 1852.716025] Key type ._llcrypt unregistered [ 1861.506856] Key type ._llcrypt registered [ 1861.507974] Key type .llcrypt registered [ 1861.548593] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 1862.013834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1862.022494] alg: No test for adler32 (adler32-zlib) [ 1862.891381] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1862.984651] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 1864.567156] Key type lgssc registered [ 1864.948283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1867.593605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1867.599511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.747290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1870.032851] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1872.926583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1873.069544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.383537] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1876.252040] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1877.409270] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1879.737225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1879.869713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.873460] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1882.984884] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 1883.853483] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1894.251457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 1897.466185] Lustre: 136760:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1898.096291] Lustre: 136809:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1898.561153] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 1899.673574] Lustre: 136958:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 1899.969887] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 1899.972348] Lustre: Skipped 1 previous similar message [ 1900.278995] Lustre: 137056:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 1903.584019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1903.586929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.591384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.609383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1904.609839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1904.609956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.612724] Lustre: Skipped 1 previous similar message [ 1907.307400] LustreError: 137154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1907.364035] Lustre: server umount lustre-OST0000 complete [ 1909.215889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1909.217950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1909.221578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.685885] LustreError: 137363:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1914.688648] LustreError: 137363:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1914.715684] Lustre: server umount lustre-MDT0000 complete [ 1914.848536] LustreError: 134488:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.854174] LustreError: 134488:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1916.020318] LustreError: 136107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677720 with bad export cookie 6846245710836444406 [ 1916.023400] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1916.029355] LustreError: 136107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1916.091265] LustreError: 137564:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1916.094203] LustreError: 137564:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1916.170665] Lustre: server umount lustre-MDT0001 complete [ 1919.789766] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 1920.898445] Key type lgssc unregistered [ 1921.045455] LNet: 138095:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.049270] LNetError: 138095:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.057398] LNet: Removed LNI 192.168.201.144@tcp [ 1921.379760] Key type .llcrypt unregistered [ 1921.380839] Key type ._llcrypt unregistered [ 1930.032123] Key type ._llcrypt registered [ 1930.033539] Key type .llcrypt registered [ 1930.074867] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 1930.450398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.467895] alg: No test for adler32 (adler32-zlib) [ 1931.336512] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1931.427570] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 1933.023156] Key type lgssc registered [ 1933.480202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1936.262481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1936.271331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.429709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.735398] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1941.526707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1941.670251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1943.044758] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1944.828915] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1945.975909] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1948.183341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1948.314060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.202695] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 1951.398480] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 1951.914808] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1962.664552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 1969.652469] Lustre: *** cfs_fail_loc=159, val=0*** [ 1984.958154] Lustre: lustre-MDT0000: Client dbd0c8e8-18ec-4ea5-84fd-6d23cf9f056f (at 192.168.201.44@tcp) reconnecting [ 2002.911939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2002.913819] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.916832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.937247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.937647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.937743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.945719] Lustre: Skipped 1 previous similar message [ 2007.341055] LustreError: 141869:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2007.400644] Lustre: server umount lustre-OST0000 complete [ 2009.057445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.058548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.061098] Lustre: Skipped 1 previous similar message [ 2014.176796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.178664] Lustre: Skipped 2 previous similar messages [ 2014.782128] LustreError: 142071:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2014.784026] LustreError: 142071:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2014.811699] Lustre: server umount lustre-MDT0000 complete [ 2016.057169] LustreError: 139449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677820 with bad export cookie 4603258875068435932 [ 2016.059888] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2016.059889] LustreError: 139449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2016.123692] LustreError: 142274:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2016.125266] LustreError: 142274:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2016.202390] Lustre: server umount lustre-MDT0001 complete [ 2019.713861] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 2020.739672] Key type lgssc unregistered [ 2020.871427] LNet: 142805:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.875162] LNetError: 142805:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2020.888292] LNet: Removed LNI 192.168.201.144@tcp [ 2021.189150] Key type .llcrypt unregistered [ 2021.190126] Key type ._llcrypt unregistered [ 2030.469724] Key type ._llcrypt registered [ 2030.471411] Key type .llcrypt registered [ 2030.503654] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 2030.846922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2030.886846] alg: No test for adler32 (adler32-zlib) [ 2031.737652] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2031.824935] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 2033.399267] Key type lgssc registered [ 2033.849387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2036.474021] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2036.481765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.667265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.026723] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2042.114811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2042.256823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.584663] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2045.326839] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2046.602498] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2048.823016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2048.936975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.763260] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2052.006913] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 2052.571285] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2062.252725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 2076.029358] Lustre: lustre-OST0000: evicting bcddd598-6559-48a3-98b8-a76348f38741 (at 192.168.201.44@tcp) by administrative request [ 2076.031842] Lustre: Skipped 1 previous similar message [ 2103.265539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2103.266988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.267496] LustreError: Skipped 1 previous similar message [ 2103.271564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2103.273831] Lustre: Skipped 1 previous similar message [ 2106.091879] LustreError: 146979:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2106.167439] Lustre: server umount lustre-OST0000 complete [ 2108.384716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.385360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.388956] Lustre: Skipped 2 previous similar messages [ 2113.504171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2113.505701] Lustre: Skipped 2 previous similar messages [ 2113.530064] LustreError: 147182:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2113.531611] LustreError: 147182:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2113.559482] Lustre: server umount lustre-MDT0000 complete [ 2114.829619] LustreError: 144159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677919 with bad export cookie 4588620114238585287 [ 2114.832380] LustreError: 144159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2114.832644] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2114.900082] LustreError: 147383:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2114.902651] LustreError: 147383:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2114.999243] Lustre: server umount lustre-MDT0001 complete [ 2118.978853] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 2120.072139] Key type lgssc unregistered [ 2120.215352] LNet: 147914:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.217387] LNetError: 147914:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.225335] LNet: Removed LNI 192.168.201.144@tcp [ 2120.550131] Key type .llcrypt unregistered [ 2120.551162] Key type ._llcrypt unregistered [ 2146.267279] Key type ._llcrypt registered [ 2146.268365] Key type .llcrypt registered [ 2146.304856] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 2146.733299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2146.741622] alg: No test for adler32 (adler32-zlib) [ 2147.592068] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2147.674898] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 2149.255165] Key type lgssc registered [ 2149.637591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2152.147100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2152.153331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2153.309676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.528295] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2157.265784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2157.396956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.774169] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2160.599268] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2161.806260] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2164.142036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2164.264588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.077906] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2167.335202] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 2167.866901] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2179.626533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2187.743959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2187.748063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2187.754548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.794493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.795058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.795386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2193.258102] LustreError: 152630:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2193.319927] Lustre: server umount lustre-OST0000 complete [ 2194.912712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.913443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.916671] Lustre: Skipped 1 previous similar message [ 2194.919386] Lustre: Skipped 2 previous similar messages [ 2200.032392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.035257] Lustre: Skipped 1 previous similar message [ 2200.635737] LustreError: 152833:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2200.637716] LustreError: 152833:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2200.663270] Lustre: server umount lustre-MDT0000 complete [ 2201.905614] LustreError: 151980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678006 with bad export cookie 6939091091951002346 [ 2201.908623] LustreError: 151980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2201.908780] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.971693] LustreError: 153034:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2201.975086] LustreError: 153034:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2202.057897] Lustre: server umount lustre-MDT0001 complete [ 2206.039619] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 2207.070435] Key type lgssc unregistered [ 2207.210431] LNet: 153565:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.212567] LNetError: 153565:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.223320] LNet: Removed LNI 192.168.201.144@tcp [ 2207.521541] Key type .llcrypt unregistered [ 2207.523858] Key type ._llcrypt unregistered [ 2216.673481] Key type ._llcrypt registered [ 2216.675065] Key type .llcrypt registered [ 2216.716030] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 2217.028914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.080056] alg: No test for adler32 (adler32-zlib) [ 2217.944603] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2218.032141] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 2219.607122] Key type lgssc registered [ 2219.970201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2222.248549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2224.536247] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 18:07:08 (1756678028) [ 2229.383970] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 2232.223153] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 2235.420382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2237.537514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2239.471745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2241.454886] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2243.625045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2243.647651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2243.653177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.731772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2244.745821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2244.783372] Lustre: lustre-MDT0000: new disk, initializing [ 2244.807228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.813513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2246.060136] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2250.077165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2250.101269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2250.141894] Lustre: 157758:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2250.169493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2250.170989] Lustre: Skipped 1 previous similar message [ 2250.205866] Lustre: lustre-MDT0001: new disk, initializing [ 2250.239458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2250.252215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2250.255965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2251.575520] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2254.608568] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2255.832943] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2258.154609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2258.181743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2258.284514] Lustre: lustre-OST0000: new disk, initializing [ 2258.286292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2258.311656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.085514] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2263.005346] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2265.078633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2265.081558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2265.113909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2275.298441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.298721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2275.302686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2275.304205] LustreError: Skipped 1 previous similar message [ 2275.309233] Lustre: Skipped 1 previous similar message [ 2278.638080] LustreError: 160343:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2278.694746] Lustre: server umount lustre-OST0000 complete [ 2280.416882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2280.417356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2280.419777] Lustre: Skipped 2 previous similar messages [ 2285.536147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2285.538198] Lustre: Skipped 2 previous similar messages [ 2290.656142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.658007] Lustre: Skipped 1 previous similar message [ 2294.239089] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2294.267325] LustreError: 160544:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2294.268951] LustreError: 160544:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2294.295683] Lustre: server umount lustre-MDT0000 complete [ 2295.424098] LustreError: 158538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678099 with bad export cookie 6713251620881131321 [ 2295.426495] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.426880] LustreError: 158538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2295.483136] LustreError: 160750:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2295.484710] LustreError: 160750:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2295.560362] Lustre: server umount lustre-MDT0001 complete [ 2298.887442] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 2299.951894] Key type lgssc unregistered [ 2300.084327] LNet: 161281:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.086257] LNetError: 161281:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.097302] LNet: Removed LNI 192.168.201.144@tcp [ 2300.396062] Key type .llcrypt unregistered [ 2300.397092] Key type ._llcrypt unregistered [ 2309.035115] Key type ._llcrypt registered [ 2309.036799] Key type .llcrypt registered [ 2309.067936] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 2309.467783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.491428] alg: No test for adler32 (adler32-zlib) [ 2310.332689] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2310.404947] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 2311.983150] Key type lgssc registered [ 2312.306825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.870142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2314.877194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.022241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.273477] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2319.965310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2320.094197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2321.352568] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2323.033185] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2324.182270] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2326.390371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2326.511774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2328.333756] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2329.973571] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2330.600110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2338.244663] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2338.254325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.261459] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2338.264839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.269660] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2338.586583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.885123] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2338.888508] Lustre: Skipped 1 previous similar message [ 2338.893392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.897760] Lustre: Skipped 1 previous similar message [ 2338.898938] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2338.900409] Lustre: Skipped 1 previous similar message [ 2338.901550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2338.904319] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2338.905820] Lustre: Skipped 1 previous similar message [ 2339.180423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2339.459045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2339.733355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2340.021664] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2340.023420] Lustre: Skipped 3 previous similar messages [ 2340.028133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.031090] Lustre: Skipped 3 previous similar messages [ 2340.032262] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2340.033791] Lustre: Skipped 3 previous similar messages [ 2340.034934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2340.039284] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2340.043034] Lustre: Skipped 3 previous similar messages [ 2340.312331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2340.608809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2340.898429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2341.215906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2341.515651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2341.809429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2342.095787] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2342.097875] Lustre: Skipped 6 previous similar messages [ 2342.102557] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.107414] Lustre: Skipped 6 previous similar messages [ 2342.110091] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2342.113393] Lustre: Skipped 6 previous similar messages [ 2342.115475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2342.119369] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2342.120821] Lustre: Skipped 6 previous similar messages [ 2342.416466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2342.704515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2343.005810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2343.287650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2343.586542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2343.868977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2344.179473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2344.466529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2344.770849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2345.052806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2345.343560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2345.627097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2345.927829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2346.233810] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2346.235529] Lustre: Skipped 13 previous similar messages [ 2346.239756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.242648] Lustre: Skipped 13 previous similar messages [ 2346.244087] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2346.246153] Lustre: Skipped 13 previous similar messages [ 2346.247394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2346.250718] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2346.254093] Lustre: Skipped 13 previous similar messages [ 2346.542707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2346.827045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2347.138726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2347.432313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2347.729919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2348.028575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2348.333751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2348.612231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2348.898824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2349.191678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2349.495946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2349.777070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2350.068098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2350.359312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2350.653946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2350.935909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2351.210655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2351.504294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2351.807696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2352.099192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2352.391183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2352.671610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2352.982928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2353.269406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2353.563072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2353.871087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2354.173368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2354.446445] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2354.448039] Lustre: Skipped 27 previous similar messages [ 2354.452459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.455407] Lustre: Skipped 27 previous similar messages [ 2354.456626] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2354.459878] Lustre: Skipped 27 previous similar messages [ 2354.462366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2354.467143] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2354.469736] Lustre: Skipped 27 previous similar messages [ 2354.761614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2355.051062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2355.346249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2355.635443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2355.934598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2356.223975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2356.534905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2356.835809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2357.133129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2357.411367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2357.705621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2358.003767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2358.301484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2358.619785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2358.934965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2359.236733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2359.537494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2359.848336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2360.178853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2360.524794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2360.849840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2361.173946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2361.513320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2361.821470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2362.122950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2362.422923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2362.742372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2363.055882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2363.374430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2363.657540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2363.947805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2364.223992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2364.513488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2364.795325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2365.094277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2365.378324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2365.659290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2365.952726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2366.265752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2366.575833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2366.900353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2367.210269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2367.552354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2367.870875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2368.163524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2368.451265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2368.762651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2369.053353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2369.325853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2369.609489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2369.916555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2370.226372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2370.527409] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2370.529948] Lustre: Skipped 52 previous similar messages [ 2370.534919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.537884] Lustre: Skipped 52 previous similar messages [ 2370.539302] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2370.541846] Lustre: Skipped 52 previous similar messages [ 2370.543746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2370.547391] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2370.549365] Lustre: Skipped 52 previous similar messages [ 2370.828681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2371.142051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2371.425626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2371.726193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2372.024943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2372.332048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2372.615526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2372.903599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2373.198527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2373.509865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2373.805146] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2374.102457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2374.386249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2374.691337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2374.993660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2375.316501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2375.620223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2375.934436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2376.221747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2376.519994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2376.824889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2377.134382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2377.432858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2377.736444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2378.050822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2378.356173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2378.664944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2378.972614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2379.266275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2379.568431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2379.857183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2380.157361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2380.447463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2380.751039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2381.050503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2381.344988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2381.641087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2381.949132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2382.240069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2382.529650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2382.820269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2383.147213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2383.454560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2383.749132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2384.037620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2384.338356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2384.631130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2384.932690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2385.252153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2385.576944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2385.883286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2386.201550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2386.505189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2386.831101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2387.138160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2387.437041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2387.743832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2388.044025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2388.333181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2388.623510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2388.934687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2389.271231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2389.563706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2389.856148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2390.159744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2390.466499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2390.760829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2391.049303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2391.334251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2391.652280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2391.961025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2392.284538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2392.622713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2392.975600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2393.299135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2393.629538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2393.918382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2394.229268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2394.520515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2394.826216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2395.125846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2395.435593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2395.731445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2396.028902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2396.311822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2396.608233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2396.894960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2397.181723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2397.472558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2397.769169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2398.057600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2398.366455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2398.667215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2398.970057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2399.255653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2399.564463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2399.855989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2400.166244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2400.471951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2400.767581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2401.064105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2401.374599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2401.657924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2401.954533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2402.239996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2402.539652] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2402.542514] Lustre: Skipped 105 previous similar messages [ 2402.548421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2402.552559] Lustre: Skipped 105 previous similar messages [ 2402.554372] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2402.556729] Lustre: Skipped 105 previous similar messages [ 2402.557937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2402.560724] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2402.562174] Lustre: Skipped 105 previous similar messages [ 2402.853855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2403.148202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2403.444235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2403.745121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2404.047344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2404.351601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2404.651872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2404.957712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2405.247577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2405.537189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2405.826559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2406.133861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2406.441579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2406.761598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2407.066915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2407.408705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2407.723227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2408.039780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2408.340560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2408.647455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2408.931336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2409.209780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2409.496799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2409.788414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2410.078180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2410.377310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2410.673787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2410.990606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2411.289661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2411.587293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2411.886454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2412.185985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2412.476060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2412.766388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2413.086489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2413.421934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2413.714049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2414.009876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2414.332258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2414.652118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2414.950768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2415.240222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2415.545133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2415.853205] LustreError: 162687:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2415.856148] LustreError: 162687:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2415.857353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2416.153335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2416.470802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2416.764630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.073558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.359778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.659937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2417.965653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2418.292640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2418.598569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2418.908397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2419.215950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2419.537222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2419.834224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2420.136429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2420.422882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2420.720211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2420.996159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2421.283312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2421.566422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2421.855650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2422.139419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2422.433231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2422.718216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2423.027739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2423.327859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2423.620938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2423.923521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2424.225856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2424.508057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2424.795351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2425.094064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2425.398315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2425.677826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2425.969903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2426.271415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2426.574573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2426.870569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2427.179693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2427.473572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2427.785099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2428.074099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2428.388103] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2428.671255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2428.979230] LustreError: 162687:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2428.983369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2428.983923] LustreError: 162687:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2429.278655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2429.573891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2429.876608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2430.196902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2430.507865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2430.812707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2431.113603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2431.429832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2431.749901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.052762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.361382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.678362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2432.978777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2433.279780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2433.579765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2433.897058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2434.191051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2434.494110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2434.784046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2435.093769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2435.394594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2435.691241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2435.982205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2436.283690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2436.572288] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2436.869797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2437.156190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2437.472514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2437.768625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2438.075386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2438.375276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2438.688415] LustreError: 162687:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -108 [ 2438.691717] LustreError: 162687:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -108 [ 2438.692414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2439.000156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2439.292495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2439.594030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2439.903253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2440.206403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2440.509978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2440.815591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2441.150975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2441.458310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2441.763459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2442.076078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2442.383777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2442.684210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2442.977495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2443.280953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2443.589722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2443.896410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2444.195863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2444.486499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2444.796256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2445.111379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2445.423992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2445.720593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2446.026208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2446.338947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2446.661857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2446.971276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2447.278415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2447.572197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2447.872473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.172787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.480155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2448.766988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2449.049415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2449.335762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2449.659152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2449.947805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2450.234057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2450.531661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2450.829728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2451.124779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2451.418987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2451.706480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2452.005591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2452.299645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2452.614979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2452.916483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2453.220744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2453.514520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2453.797701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2454.082783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2454.394511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2454.804128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2455.128494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2455.486123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2455.823239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2456.098156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2456.379698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2456.679983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2456.957210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2457.248920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2457.547803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2457.852924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2458.181319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2458.582768] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2458.903493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2459.202021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2459.507022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2459.814042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2460.111268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2460.382896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2460.661063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2460.955310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2461.236064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2461.511713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2461.796431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2462.089558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2462.390036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2462.698071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2463.041299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2463.376423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2463.675963] LustreError: 162687:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2463.678738] LustreError: 162687:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2463.680363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2463.977262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2464.269827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2464.559625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2464.866567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2465.159885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2465.443496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2465.741538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2466.036112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2466.313723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2466.585417] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2466.586977] Lustre: Skipped 211 previous similar messages [ 2466.591282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.594267] Lustre: Skipped 211 previous similar messages [ 2466.595529] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2466.597263] Lustre: Skipped 211 previous similar messages [ 2466.598404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2466.601192] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2466.603380] Lustre: Skipped 211 previous similar messages [ 2466.869245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2467.157104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2467.455317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2467.758075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2468.132156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2468.479903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2468.826496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2469.168922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2469.522778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2469.827841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2470.122722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2470.412983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2470.756844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2471.084065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2471.386721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2471.689735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2472.000856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2472.297936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2472.585894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2472.878141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2473.175499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2473.468754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2473.774246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2474.086005] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2474.397920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2474.704739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2475.017452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2475.309500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2475.621551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2475.918861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2476.220212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2476.537493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2476.835244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2477.143739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2477.442364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2477.733316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2478.038189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2478.319298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2478.607327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2478.906387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2479.205387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2479.533147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2479.910370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2480.224771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2480.534867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2480.824326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2481.155058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2481.437832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2481.747529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2482.041296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2482.380498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2482.799379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2483.103638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2483.411066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2483.693537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2483.994094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2484.282442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2484.585661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2484.888610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2485.178584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2485.473065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2485.764228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2486.062453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2486.353867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2486.638573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2486.927573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2487.227496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2487.513744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2487.807115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.090680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.393789] LustreError: 162687:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2488.397608] LustreError: 162687:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2488.398321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.687903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2488.975910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2489.276499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2489.572802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2489.865453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2490.155356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2490.437476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2490.732427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2491.009365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2491.287457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2491.570399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2491.871129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2492.163536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2492.440549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2492.724454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2493.004251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2493.282865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2493.565515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2493.851479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2494.143417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2494.442953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2494.758843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2495.087227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2495.444118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2495.858382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2496.178150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2496.492798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2496.814800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2497.127807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2497.419692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2497.712208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2498.022613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2498.331897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2498.645338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2498.966652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2499.310926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2499.611735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2499.924922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2500.216039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2500.514040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2500.812197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2501.113097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2501.423407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2501.720702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.005731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.312822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.594684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2502.875299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.157237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.451277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2503.732835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.010113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.314076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.622674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2504.944794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.279653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.595783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2505.913320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.201563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.492373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2506.790954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.099315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.396839] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.678398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2507.963849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.250755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.538940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2508.827176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.123522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.423978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2509.722902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.016174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.326642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.624915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2510.908683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.198237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.482602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2511.767035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.057158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.347987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.633338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2512.924916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.214806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.513318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2513.816211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.129311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.426281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2514.726795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.026314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.324751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.619517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2515.900758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.204198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.506286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2516.793438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.090549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.384497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.695152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2517.992230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.290186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.574408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2518.890087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.192713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.497582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2519.797350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.152782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.466374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2520.769176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.076816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.394095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.695553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2521.989794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.286980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.579669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2522.858126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.143802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.444377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2523.739382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.028760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.319301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.615788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2524.917840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.208603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.486530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2525.784443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.069508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.361252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.650957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2526.931775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.231108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.508308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2527.787855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.088094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.390894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.680717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2528.965975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.256440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.552290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2529.835724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.113079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.401334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.689084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2530.999013] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.286501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.579670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2531.913365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.221641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.520476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2532.812694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.111169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.388624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.681548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2533.986499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.276563] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.556550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2534.836770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.118639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.418567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.710904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2535.995060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.288148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.603136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2536.901338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.180782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.462832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2537.763674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.050703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.330347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.615211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2538.908307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.183365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.454925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2539.734169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.021901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.306523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.586426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2540.858835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.157954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.446206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2541.726954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.017976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.325173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.634026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2542.918936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.212870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.521655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2543.827933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.138887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.429418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2544.764492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.069688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.375652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.683797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2545.999939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.292393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.591180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2546.896415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.213731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.532099] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2547.853216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.152830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.488570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2548.795603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.093280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.396865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2549.728645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.040161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.330691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.624433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2550.929019] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.220092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.508278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2551.808889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.114077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.403061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2552.700580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.000051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.306616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.609331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2553.907200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.194794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.504276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2554.790732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.079670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.370956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.662896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2555.955875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.240979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.538369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2556.836790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.138515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.430663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2557.731138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.049371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.338807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.631380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2558.932342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.245244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.551926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2559.850129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.136338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.431143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2560.720256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.014170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.301745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.597863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2561.876764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.165766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.445874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2562.747374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.048614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.338232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.624595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2563.939292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.240174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.537132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2564.849413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.157359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.457302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2565.747476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.042784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.367223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.684327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2566.992213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.301319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.596208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2567.875849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.168391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.451267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2568.765580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.054994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.330986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.605721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2569.894620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.202559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.505508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2570.812321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.115875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.417859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2571.701517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.011669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.334409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.644394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2572.942906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2573.239746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2573.552952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2573.858074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.162861] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.455657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2574.757902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.054313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.336255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.619944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2575.916274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2576.200292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2576.490603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2576.787625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.090253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.380404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.674494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2577.963614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.263223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.564764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2578.852922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.138939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.437894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2579.738868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.030267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.318644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.619516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2580.903387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2581.207904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2581.521387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2581.830365] LustreError: 162687:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2581.833027] LustreError: 162687:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2581.834388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.129790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.431511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2582.740440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.072951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.375539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.665095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2583.949036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.254790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.557698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2584.843412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2585.136445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2585.431736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2585.717456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.018148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.315810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.635302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2586.935352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.228236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.527350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2587.830532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.139678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.441855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2588.747055] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.058326] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.351095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.645810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2589.929574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.237940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.525279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2590.822714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.106143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.404870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.686833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2591.976632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2592.270182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2592.602597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2592.963305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.292559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.592806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2593.917546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.224387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.525192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.803606] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2594.805500] Lustre: Skipped 427 previous similar messages [ 2594.811638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.816108] Lustre: Skipped 427 previous similar messages [ 2594.817374] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2594.819162] Lustre: Skipped 427 previous similar messages [ 2594.820370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2594.824164] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2594.826282] Lustre: Skipped 427 previous similar messages [ 2595.134222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2595.431135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2595.725181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.001373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.295125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.584613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2596.861371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.148059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.452317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2597.754120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.038867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.316205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.604956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2598.889926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.170168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.467600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2599.753340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.032121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.322067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.612397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2600.918951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.203913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.504380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2601.821207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.133614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.436193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2602.741482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.028892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.327826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.610519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2603.898569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.185527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.486754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2604.766323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.049423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.343838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.633735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2605.913511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.198375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.483274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2606.785331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.079012] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.363720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.653709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2607.958533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.267732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.568805] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2608.859477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.168769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.468221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2609.772638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.074989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.391956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.685064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2610.960763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.250027] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.536079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2611.838821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.149532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.443145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2612.739049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.028406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.306294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.595727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2613.900955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.196912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.485174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2614.766873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.059346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.351281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.638439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2615.921814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.203158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.485269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2616.765250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.047012] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.341162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.622909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2617.911040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.190522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.486132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2618.780665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.062643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.354455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.670573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2619.968229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.250542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.535353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.818865] LustreError: 162687:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 2620.823425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2620.824259] LustreError: 162687:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 2621.125664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.424808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.719930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2621.998549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.289280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2622.573244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2648.035270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2648.035786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2651.754089] LustreError: 213801:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2651.840425] Lustre: server umount lustre-OST0000 complete [ 2653.155560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2653.156429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2653.159232] Lustre: Skipped 2 previous similar messages [ 2658.272072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2658.274518] Lustre: Skipped 1 previous similar message [ 2659.205587] LustreError: 214002:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2659.208883] LustreError: 214002:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2659.257855] Lustre: server umount lustre-MDT0000 complete [ 2660.485039] LustreError: 162649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678464 with bad export cookie 2559792288098092483 [ 2660.488747] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2660.554031] LustreError: 214204:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2660.556674] LustreError: 214204:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2660.635315] Lustre: server umount lustre-MDT0001 complete [ 2664.192046] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 2665.238719] Key type lgssc unregistered [ 2665.385332] LNet: 214736:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2665.388524] LNetError: 214736:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2665.395289] LNet: Removed LNI 192.168.201.144@tcp [ 2665.726522] Key type .llcrypt unregistered [ 2665.728154] Key type ._llcrypt unregistered [ 2674.589661] Key type ._llcrypt registered [ 2674.590667] Key type .llcrypt registered [ 2674.624837] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 2674.964845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2674.999803] alg: No test for adler32 (adler32-zlib) [ 2675.847877] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2675.932430] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 2677.519128] Key type lgssc registered [ 2677.868994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2680.360650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2680.366924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.529951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.742740] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2685.470748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2685.582681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.828907] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2688.462667] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2689.588425] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2691.716930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2691.836441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.693726] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2695.380969] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2706.217538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:16947 to 0x280000401:16993) [ 2713.965948] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 2714.591965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.596012] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2714.598175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2714.600827] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2714.600855] LustreError: 216126:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 2715.934228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2715.935444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.936562] LustreError: Skipped 1 previous similar message [ 2715.942719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.641979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2716.643815] Lustre: Skipped 2 previous similar messages [ 2717.995184] LustreError: 218525:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2718.052806] Lustre: server umount lustre-OST0000 complete [ 2734.607086] LustreError: 216126:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 2746.005139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2746.116733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2747.962887] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2749.728673] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2757.397932] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec [ 2757.412244] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2757.416065] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2758.503827] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92c5a0c70800.ost_server_uuid 50 [ 2758.967602] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92c5a0c70800.ost_server_uuid in FULL state after 0 sec [ 2762.723061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.723609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2762.726407] Lustre: Skipped 1 previous similar message [ 2762.728813] LustreError: Skipped 1 previous similar message [ 2762.731802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.996764] LustreError: 219842:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2765.998597] LustreError: 219842:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2766.068127] Lustre: server umount lustre-OST0000 complete [ 2767.840048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2767.842492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.846630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.936400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2771.936818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.434718] LustreError: 220043:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2773.436426] LustreError: 220043:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2773.461628] Lustre: server umount lustre-MDT0000 complete [ 2774.670065] LustreError: 216088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678579 with bad export cookie 15744456629759891156 [ 2774.673273] LustreError: 216088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2774.673395] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2774.807366] Lustre: server umount lustre-MDT0001 complete [ 2778.378553] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 2779.522771] Key type lgssc unregistered [ 2779.658361] LNet: 220776:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2779.660605] LNetError: 220776:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2779.669295] LNet: Removed LNI 192.168.201.144@tcp [ 2779.970607] Key type .llcrypt unregistered [ 2779.971623] Key type ._llcrypt unregistered [ 2791.863770] Key type ._llcrypt registered [ 2791.865405] Key type .llcrypt registered [ 2791.906632] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 2792.319086] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2792.353951] alg: No test for adler32 (adler32-zlib) [ 2793.221987] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2793.323849] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 2794.911143] Key type lgssc registered [ 2795.275133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2798.638298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2798.647446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.841761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2801.379329] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2804.943713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2805.074245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2806.351736] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2807.370392] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2809.496188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2809.612753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.664390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:16995 to 0x280000401:17025) [ 2812.593220] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2815.964386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2816.033056] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2816.092949] Lustre: lustre-OST0001: new disk, initializing [ 2816.094619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2816.127196] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2817.976782] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2823.166128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2823.169585] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2823.206330] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2823.226559] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2826.730755] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2841.056091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2841.060494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2841.067243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.616671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.619681] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.622534] Lustre: Skipped 2 previous similar messages [ 2846.464512] LustreError: 225826:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2846.491990] Lustre: server umount lustre-MDT0000 complete [ 2847.664370] LustreError: 225535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678652 with bad export cookie 12167095736561892527 [ 2847.667221] LustreError: 225535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2847.667802] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2847.729612] LustreError: 226027:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2847.732691] LustreError: 226027:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2847.809630] Lustre: server umount lustre-MDT0001 complete [ 2859.250105] LustreError: 226227:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2859.253283] LustreError: 226227:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2859.270611] Lustre: server umount lustre-OST0000 complete [ 2870.643173] LustreError: 226430:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2870.645599] LustreError: 226430:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2870.737361] Lustre: server umount lustre-OST0001 complete [ 2871.831437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2871.860389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2872.233737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2872.275526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2872.640626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2872.669474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2873.038714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2873.067167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2878.123751] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 2881.664102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.868309] LustreError: 227839:0:(ldlm_lib.c:1132:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.877039] LustreError: 227839:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2881.890578] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2883.181699] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2886.027849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2887.408094] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2888.325973] Lustre: 228947:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2890.461298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2890.605750] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.607425] Lustre: Skipped 1 previous similar message [ 2892.435747] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2892.642943] LustreError: 229302:0:(ldlm_lib.c:1132:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2892.648069] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:17027 to 0x280000401:17057) [ 2895.218788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2896.299463] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2897.028640] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2899.989914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2901.056801] Lustre: 230793:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2911.716322] Lustre: mylustre-MDT0000-lwp-OST0000: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2911.717965] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2911.718243] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2911.718250] Lustre: Skipped 3 previous similar messages [ 2911.722342] Lustre: Skipped 2 previous similar messages [ 2916.414256] LustreError: 231100:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2916.415957] LustreError: 231100:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2916.446703] Lustre: server umount mylustre-MDT0000 complete [ 2916.833556] LustreError: 227838:0:(ldlm_lib.c:1132:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.839954] LustreError: 227838:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2917.635587] LustreError: 227813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678721 with bad export cookie 12167095736561904042 [ 2917.642728] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2917.806864] Lustre: server umount mylustre-MDT0001 complete [ 2929.139097] LustreError: 231502:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 2929.140915] LustreError: 231502:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 2929.155891] Lustre: server umount mylustre-OST0000 complete [ 2940.655292] Lustre: server umount mylustre-OST0001 complete [ 2941.678096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2941.705054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2942.038546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 2942.068512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2942.424479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2942.458703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2942.828790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 2942.868113] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2947.157543] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 2950.534666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2950.685312] LustreError: 233112:0:(ldlm_lib.c:1132:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.708472] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2950.710925] Lustre: Skipped 1 previous similar message [ 2951.986416] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2954.786351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2956.201855] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2957.154913] Lustre: 234223:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2959.354516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2961.326401] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2961.510437] LustreError: 234576:0:(ldlm_lib.c:1132:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.515878] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:17027 to 0x280000401:17089) [ 2964.125278] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2965.226021] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 2966.134316] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 2969.312251] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2970.500753] Lustre: 236065:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2980.834487] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2980.836218] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.838021] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 2980.838805] Lustre: Skipped 1 previous similar message [ 2980.840917] Lustre: Skipped 3 previous similar messages [ 2983.425090] LustreError: 236316:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2983.427058] LustreError: 236316:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2983.463774] Lustre: server umount tfs-MDT0000 complete [ 2984.672401] LustreError: 233841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678789 with bad export cookie 12167095736561905932 [ 2984.675938] LustreError: 233841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2984.676057] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2984.812241] Lustre: server umount tfs-MDT0001 complete [ 2996.220779] Lustre: server umount tfs-OST0000 complete [ 3007.610335] Lustre: server umount tfs-OST0001 complete [ 3008.593778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3008.621931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3008.968977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 3008.995718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3009.341226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3009.365765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3009.702947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 3009.730517] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3014.022630] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3017.264316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.411241] LustreError: 238326:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3017.434613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.438195] Lustre: Skipped 3 previous similar messages [ 3018.694555] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3021.364757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3022.752633] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3023.676905] Lustre: 239438:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3025.773678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3026.918760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:17027 to 0x280000401:17121) [ 3027.662840] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3030.417067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3031.532275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3032.267785] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3035.342497] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3036.584947] Lustre: 241283:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3043.727293] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 18:20:47 (1756678847) [ 3057.123372] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.125754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3057.127856] Lustre: Skipped 4 previous similar messages [ 3057.129275] Lustre: Skipped 3 previous similar messages [ 3061.697250] LustreError: 241650:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3061.700430] LustreError: 241650:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3061.732835] Lustre: server umount lustre-MDT0000 complete [ 3062.241372] LustreError: 239042:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.245322] LustreError: 239042:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 3062.893164] LustreError: 238301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678867 with bad export cookie 12167095736561907598 [ 3062.897148] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.361222] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.362216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3067.364679] Lustre: Skipped 2 previous similar messages [ 3067.367418] Lustre: Skipped 3 previous similar messages [ 3069.132755] Lustre: server umount lustre-MDT0001 complete [ 3076.476587] Lustre: server umount lustre-OST0000 complete [ 3083.758581] Lustre: server umount lustre-OST0001 complete [ 3085.807360] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 3089.029189] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3092.356836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3094.638441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3096.663529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3098.988826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3101.520220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3101.556962] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 3101.556962] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 3101.556962] [ 3101.569211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 3101.684216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3101.701645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3101.757410] Lustre: lustre-MDT0000: new disk, initializing [ 3101.796860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.801160] Lustre: Skipped 3 previous similar messages [ 3101.812357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3103.421895] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3107.777906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3107.810913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3107.849087] Lustre: 244433:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3107.875228] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3107.877418] Lustre: Skipped 1 previous similar message [ 3107.916095] Lustre: lustre-MDT0001: new disk, initializing [ 3107.961046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3107.964171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3109.274908] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3113.368443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3113.392900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3113.487412] Lustre: lustre-OST0000: new disk, initializing [ 3113.489360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3115.063245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3115.067221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3115.109848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3115.367975] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3120.136070] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3120.166107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3120.242533] Lustre: lustre-OST0001: new disk, initializing [ 3120.245060] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3121.482814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3121.486290] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3121.528572] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3122.533855] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3127.473190] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 18:22:11 (1756678931) [ 3131.086981] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 18:22:14 (1756678934) [ 3145.696238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3145.701301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.710965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3148.914179] LustreError: 247854:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3148.917622] LustreError: 247854:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3148.945279] Lustre: server umount lustre-OST0000 complete [ 3152.356848] LustreError: 245970:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.364524] LustreError: 245970:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3156.917059] Lustre: server umount lustre-MDT0000 complete [ 3158.373170] LustreError: 244424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678962 with bad export cookie 12167095736561909152 [ 3158.376776] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3158.377199] LustreError: 244424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3164.614836] Lustre: server umount lustre-MDT0001 complete [ 3168.329934] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 3186.143097] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3186.219637] Lustre: server umount lustre-OST0001 complete [ 3188.136714] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 3194.135375] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3197.437951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3199.549175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3201.448318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3203.413314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3205.586221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3205.607986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.686394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3205.698755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3205.736031] Lustre: lustre-MDT0000: new disk, initializing [ 3205.768037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3207.056420] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3210.961896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3210.987957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3211.018373] Lustre: 251315:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3211.117651] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3212.313710] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3215.257145] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3216.356220] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3218.408768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.431871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3220.024662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3220.058192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3220.299104] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3223.181773] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3230.177524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3230.178931] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.181016] LustreError: Skipped 2 previous similar messages [ 3230.181725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3230.184418] Lustre: Skipped 6 previous similar messages [ 3230.185880] Lustre: Skipped 8 previous similar messages [ 3231.528864] Lustre: server umount lustre-OST0000 complete [ 3238.806070] Lustre: server umount lustre-MDT0000 complete [ 3239.974125] LustreError: 251307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679044 with bad export cookie 12167095736561910622 [ 3239.977616] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3239.977953] LustreError: 251307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3240.119216] Lustre: server umount lustre-MDT0001 complete [ 3243.576013] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 3244.613873] Key type lgssc unregistered [ 3244.749352] LNet: 254684:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3244.752545] LNetError: 254684:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3244.764452] LNet: Removed LNI 192.168.201.144@tcp [ 3245.126152] Key type .llcrypt unregistered [ 3245.127156] Key type ._llcrypt unregistered [ 3255.349737] Key type ._llcrypt registered [ 3255.350595] Key type .llcrypt registered [ 3255.382389] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3255.767073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3255.772913] alg: No test for adler32 (adler32-zlib) [ 3256.622026] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3256.699084] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 3258.271111] Key type lgssc registered [ 3258.603248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.107085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3261.113932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.270745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3263.542439] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3266.529523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3266.665110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.383766] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3270.598453] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3272.022527] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3274.953296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3275.072455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.306602] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3279.533718] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3280.038502] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 3281.381795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.389196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3281.389700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3282.992148] LustreError: 258398:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3283.075295] Lustre: server umount lustre-OST0000 complete [ 3286.497543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.503838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3286.507783] Lustre: Skipped 1 previous similar message [ 3286.783513] LustreError: 258603:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3286.786674] LustreError: 258603:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3286.816400] Lustre: server umount lustre-MDT0000 complete [ 3288.616636] LustreError: 256085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679092 with bad export cookie 12091920264940529589 [ 3288.622975] LustreError: 256085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3288.624699] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3290.771626] LustreError: 258805:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3290.775242] LustreError: 258805:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3290.914394] Lustre: server umount lustre-MDT0001 complete [ 3295.300422] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 3296.632263] Key type lgssc unregistered [ 3296.757349] LNet: 259336:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3296.759991] LNetError: 259336:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3296.769295] LNet: Removed LNI 192.168.201.144@tcp [ 3297.108092] Key type .llcrypt unregistered [ 3297.109487] Key type ._llcrypt unregistered [ 3303.517347] Key type ._llcrypt registered [ 3303.519893] Key type .llcrypt registered [ 3303.573630] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3303.973332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3304.004068] alg: No test for adler32 (adler32-zlib) [ 3304.862499] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3304.979308] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 3306.567212] Key type lgssc registered [ 3307.111778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3310.397066] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3310.402824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3327.903364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3333.023264] LustreError: 260487:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.144@tcp: failed processing log, type 4: rc = -110 [ 3363.743235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.392277] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3368.698563] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3369.600689] LustreError: 261152:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3369.658416] Lustre: server umount lustre-OST0000 complete [ 3376.001667] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 3377.318814] Key type lgssc unregistered [ 3377.448389] LNet: 261987:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3377.454186] LNetError: 261987:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3377.466437] LNet: Removed LNI 192.168.201.144@tcp [ 3377.781532] Key type .llcrypt unregistered [ 3377.783715] Key type ._llcrypt unregistered [ 3393.493891] Key type ._llcrypt registered [ 3393.494886] Key type .llcrypt registered [ 3393.539889] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3393.949564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3393.976910] alg: No test for adler32 (adler32-zlib) [ 3394.863526] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3394.967626] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 3396.559168] Key type lgssc registered [ 3397.070145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3401.744598] loop0: detected capacity change from 0 to 204800000 [ 3401.843615] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3401.847848] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3401.969340] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3401.972670] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3401.999139] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3402.020722] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3402.039216] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3402.078977] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3402.125728] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3402.164616] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3402.479925] loop0: detected capacity change from 0 to 204800000 [ 3402.545682] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3402.870883] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3403.205937] loop0: detected capacity change from 0 to 204800000 [ 3403.393772] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3403.729311] loop0: detected capacity change from 0 to 204800000 [ 3403.855865] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3404.145953] loop0: detected capacity change from 0 to 204800000 [ 3404.153709] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3405.648798] loop0: detected capacity change from 0 to 204800000 [ 3405.666101] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3406.958396] loop0: detected capacity change from 0 to 204800000 [ 3406.969568] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3408.226304] loop0: detected capacity change from 0 to 204800000 [ 3408.233922] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 3409.555214] loop0: detected capacity change from 0 to 204800000 [ 3409.595536] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3409.634605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3409.640321] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.658924] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3409.663268] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3409.667768] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 3409.672202] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 3409.676570] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3409.681326] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 3409.685706] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3409.688764] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 3409.691464] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 3409.695913] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 3409.700509] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 3409.704906] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 3409.709087] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 3409.713370] Lustre: 265361:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 3412.387852] LustreError: 265629:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3412.426747] Lustre: server umount MGS complete [ 3412.722764] loop0: detected capacity change from 0 to 204800000 [ 3412.753509] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3412.899715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.913087] Lustre: lustre-MDT0000: Aborting client recovery [ 3413.199184] loop1: detected capacity change from 0 to 204800000 [ 3413.233823] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3413.262963] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3413.286876] Lustre: 265797:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3413.289438] Lustre: 265797:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 3413.292060] Lustre: 265797:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3413.294761] Lustre: 265797:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3413.297592] Lustre: 265797:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 3413.300244] Lustre: 265797:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 3413.302793] Lustre: 265797:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3413.308569] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3413.710241] loop2: detected capacity change from 0 to 204800000 [ 3413.751919] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3413.774498] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3413.800158] Lustre: 265881:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3413.805449] Lustre: 265881:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3413.808617] Lustre: 265881:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3413.811397] Lustre: 265881:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 3413.815604] Lustre: 265881:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 3413.818757] Lustre: 265881:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 3413.823382] Lustre: 265881:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3413.872398] Lustre: Found index 0 for lustre-OST0000, updating log [ 3413.901791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.904791] Lustre: Skipped 1 previous similar message [ 3413.911189] Lustre: lustre-OST0000: Aborting recovery [ 3414.219855] loop3: detected capacity change from 0 to 204800000 [ 3414.270981] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3414.292636] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3414.311280] Lustre: 265969:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 3414.314078] Lustre: 265969:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 3414.316876] Lustre: 265969:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 3414.319687] Lustre: 265969:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 3414.322733] Lustre: 265969:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 3414.325515] Lustre: 265969:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 3414.328339] Lustre: 265969:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 3419.626455] LustreError: 265740:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -108 [ 3422.693947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3422.695694] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 3422.705511] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 3422.710647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 3422.711281] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 3422.716073] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 3425.177976] LustreError: 265708:0:(mdt_identity.c:95:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 275us: rc = -2 [ 3438.049765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.050285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3438.057720] Lustre: Skipped 3 previous similar messages [ 3438.063376] Lustre: Skipped 3 previous similar messages [ 3442.498194] LustreError: 266332:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3442.501110] LustreError: 266332:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3442.543184] Lustre: server umount lustre-MDT0000 complete [ 3442.942760] LustreError: 265690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679247 with bad export cookie 5699663031689172757 [ 3442.945515] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3442.947680] LustreError: 265690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3443.167600] LustreError: 265705:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.167772] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3443.168438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3443.171696] LustreError: 265705:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3443.174709] Lustre: Skipped 1 previous similar message [ 3447.199513] LustreError: 265705:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3447.199713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3447.205040] LustreError: 265705:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3447.210748] Lustre: Skipped 2 previous similar messages [ 3449.087735] LustreError: 266386:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3449.089520] LustreError: 266386:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3449.184148] Lustre: server umount lustre-MDT0001 complete [ 3455.596054] LustreError: 266437:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3455.598164] LustreError: 266437:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3455.621876] Lustre: server umount lustre-OST0000 complete [ 3470.303166] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3470.324408] LustreError: 266488:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3470.328168] LustreError: 266488:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3470.437181] Lustre: server umount lustre-OST0001 complete [ 3473.590153] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 18:27:57 (1756679277) [ 3483.227828] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 3486.593528] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3490.510699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3493.043372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3495.168833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3497.215881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3500.079972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3500.119780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.225411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3500.236957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3500.271360] Lustre: lustre-MDT0000: new disk, initializing [ 3500.294172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.295816] Lustre: Skipped 1 previous similar message [ 3500.301754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3501.701977] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3506.257848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3506.280667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3506.319327] Lustre: 270034:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3506.343885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3506.345626] Lustre: Skipped 1 previous similar message [ 3506.380298] Lustre: lustre-MDT0001: new disk, initializing [ 3506.411515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3506.423029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3506.426034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3507.674531] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3510.913790] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3512.062244] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3514.305419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3514.326690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3514.424900] Lustre: lustre-OST0000: new disk, initializing [ 3514.426848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3514.457523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.289198] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3519.342163] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3522.556229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3522.560385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3522.593561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3526.703694] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3529.453499] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 3539.542981] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3546.114738] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 3578.852695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3578.853954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.857360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.861903] Lustre: Skipped 1 previous similar message [ 3578.866665] Lustre: Skipped 1 previous similar message [ 3583.969600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3583.972449] Lustre: Skipped 1 previous similar message [ 3584.367161] LustreError: 273651:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3584.370071] LustreError: 273651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3584.447943] Lustre: server umount lustre-OST0000 complete [ 3588.576119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3588.580560] LustreError: Skipped 1 previous similar message [ 3588.582294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3594.208048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3594.210279] Lustre: Skipped 4 previous similar messages [ 3600.351212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3600.394330] LustreError: 273854:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3600.397947] LustreError: 273854:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3600.446637] Lustre: server umount lustre-MDT0000 complete [ 3601.929436] LustreError: 270024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679406 with bad export cookie 5699663031689175466 [ 3601.932888] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3601.933302] LustreError: 270024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3602.087097] Lustre: server umount lustre-MDT0001 complete [ 3605.195509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.526205] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3608.228750] Lustre: server umount MGS complete [ 3612.585378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3612.744046] LustreError: 275621:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.750769] LustreError: 275633:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3612.774678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.116133] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3617.220125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3618.568362] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3620.442736] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3622.127374] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3624.712716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3627.013436] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3629.002382] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3662.816084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3662.818646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.822596] Lustre: Skipped 1 previous similar message [ 3662.824746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3662.827883] Lustre: Skipped 2 previous similar messages [ 3667.373754] LustreError: 278300:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3667.377045] LustreError: 278300:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3667.466755] Lustre: server umount lustre-OST0000 complete [ 3673.568391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3673.572630] LustreError: Skipped 1 previous similar message [ 3673.574985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.581651] Lustre: Skipped 1 previous similar message [ 3675.349287] Lustre: server umount lustre-MDT0000 complete [ 3676.602890] LustreError: 276340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679480 with bad export cookie 5699663031689177279 [ 3676.608513] LustreError: 276340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3676.608775] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3676.775818] Lustre: server umount lustre-MDT0001 complete [ 3680.424898] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 3681.584624] Key type lgssc unregistered [ 3681.710320] LNet: 279234:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3681.713955] LNetError: 279234:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3681.724298] LNet: Removed LNI 192.168.201.144@tcp [ 3682.111834] Key type .llcrypt unregistered [ 3682.113721] Key type ._llcrypt unregistered [ 3695.461864] Key type ._llcrypt registered [ 3695.462866] Key type .llcrypt registered [ 3695.495974] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 3701.911571] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3702.239761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3702.258407] alg: No test for adler32 (adler32-zlib) [ 3703.123235] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3703.216947] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 3704.799280] Key type lgssc registered [ 3705.211231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3708.346459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3710.876385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3713.021952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3715.523967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3718.080172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3718.105883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3718.115484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.209875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3719.223360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3719.263124] Lustre: lustre-MDT0000: new disk, initializing [ 3719.289933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3719.296926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3720.700262] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3725.757756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3725.786600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3725.822676] Lustre: 283138:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3725.859537] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3725.860935] Lustre: Skipped 1 previous similar message [ 3725.896286] Lustre: lustre-MDT0001: new disk, initializing [ 3725.942171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3725.965340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3725.972095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3727.574891] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3730.934574] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3732.168922] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3734.587188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3734.620566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3734.749159] Lustre: lustre-OST0000: new disk, initializing [ 3734.751060] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3734.788675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.790671] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3738.685548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3738.688794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3738.731596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3740.123093] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3743.498697] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 3748.626429] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 3753.404591] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 3760.061945] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 3800.031947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3800.034426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.039683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3800.545633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3800.546388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3800.548983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3805.422186] LustreError: 286659:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3805.508226] Lustre: server umount lustre-OST0000 complete [ 3808.224544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3808.229062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.236728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3810.784856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3810.785790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.380136] LustreError: 286860:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3813.381840] LustreError: 286860:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3813.409242] Lustre: server umount lustre-MDT0000 complete [ 3814.760645] LustreError: 283130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679619 with bad export cookie 13952191108315808056 [ 3814.764326] LustreError: 283130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3814.765081] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3814.845763] LustreError: 287062:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3814.849265] LustreError: 287062:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3814.945114] Lustre: server umount lustre-MDT0001 complete [ 3817.828473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.108459] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3820.766482] LustreError: 288003:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3820.769723] LustreError: 288003:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3820.790537] Lustre: server umount MGS complete [ 3825.160096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3825.360256] LustreError: 288637:0:(qmt_pool.c:1513:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 3825.399552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3826.863168] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3830.077712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3830.197281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3831.625945] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3833.524557] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3834.899140] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3837.937548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3840.164596] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3842.622904] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3881.442680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3881.446445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.448455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3881.453286] Lustre: Skipped 1 previous similar message [ 3881.456622] Lustre: Skipped 1 previous similar message [ 3882.542136] LustreError: 291306:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3882.544896] LustreError: 291306:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3882.632273] Lustre: server umount lustre-OST0000 complete [ 3886.564351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3886.569022] LustreError: Skipped 1 previous similar message [ 3890.425598] Lustre: server umount lustre-MDT0000 complete [ 3891.681091] LustreError: 289339:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.690099] LustreError: 289339:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3892.042806] LustreError: 288605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679696 with bad export cookie 13952191108315809995 [ 3892.047713] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3892.047861] LustreError: 288605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3892.143593] LustreError: 291709:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3892.147083] LustreError: 291709:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3892.243843] Lustre: server umount lustre-MDT0001 complete [ 3896.417605] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 3897.828445] Key type lgssc unregistered [ 3897.965653] LNet: 292241:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3897.969551] LNetError: 292241:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3897.978553] LNet: Removed LNI 192.168.201.144@tcp [ 3898.397235] Key type .llcrypt unregistered [ 3898.398221] Key type ._llcrypt unregistered [ 3905.403021] Key type ._llcrypt registered [ 3905.404272] Key type .llcrypt registered [ 3905.445571] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 3911.486661] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3911.769714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3911.775191] alg: No test for adler32 (adler32-zlib) [ 3912.634923] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3912.727228] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 3914.303177] Key type lgssc registered [ 3914.907960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3918.342226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3920.642345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3923.980002] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 3926.801610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3929.725463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3929.759167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3929.768656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.875360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3930.891367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3930.927407] Lustre: lustre-MDT0000: new disk, initializing [ 3930.960872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.969621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3932.852889] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3937.550166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3937.579780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3937.709713] Lustre: lustre-OST0000: new disk, initializing [ 3937.711784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3937.713532] Lustre: Skipped 1 previous similar message [ 3937.745844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3939.961446] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 3940.440375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3940.449029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3940.462358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 4137.953314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.959766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4141.457259] LustreError: 297156:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 4141.634413] Lustre: server umount lustre-MDT0000 complete [ 4143.364823] LustreError: 295404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679947 with bad export cookie 11789726932305219440 [ 4143.368977] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4143.372449] LustreError: 295404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4143.402918] LustreError: 297361:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4143.406343] LustreError: 297361:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4143.497936] Lustre: server umount lustre-OST0000 complete [ 4152.958029] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 4156.607931] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 4160.322383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4162.481713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4165.006863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4167.156666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4170.468281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4170.502154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4170.619060] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4170.634924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4170.682955] Lustre: lustre-MDT0000: new disk, initializing [ 4170.709733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4170.716788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4171.983460] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4176.774872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4176.797962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4176.828411] Lustre: 300398:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4176.851360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4176.853095] Lustre: Skipped 1 previous similar message [ 4176.888494] Lustre: lustre-MDT0001: new disk, initializing [ 4176.918689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4176.930857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4176.933535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4178.231143] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4181.490537] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4182.594965] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4184.853291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4184.879253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4184.971064] Lustre: lustre-OST0000: new disk, initializing [ 4184.973137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4185.001994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.828650] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4187.399168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4187.406742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4187.453092] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4190.346343] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4191.676940] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4191.743268] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4193.081397] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4193.160572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4197.857111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4197.857925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4197.859499] LustreError: Skipped 1 previous similar message [ 4197.870704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4197.872548] Lustre: Skipped 1 previous similar message [ 4202.980613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4202.984582] Lustre: Skipped 1 previous similar message [ 4208.095156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4208.102632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4208.109292] Lustre: Skipped 1 previous similar message [ 4208.130234] LustreError: 303338:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4208.135288] LustreError: 303338:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4208.244928] Lustre: server umount lustre-OST0000 complete [ 4212.703851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4212.705767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.708522] Lustre: Skipped 1 previous similar message [ 4212.709955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.867826] LustreError: 303540:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4215.869503] LustreError: 303540:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4215.894336] Lustre: server umount lustre-MDT0000 complete [ 4217.158155] LustreError: 300389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680021 with bad export cookie 11789726932308160217 [ 4217.161927] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4217.297849] Lustre: server umount lustre-MDT0001 complete [ 4218.553788] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 4219.163747] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 18:40:22 (1756680022) [ 4221.762056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.947616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4223.151704] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4226.386478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4227.937020] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4229.817716] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4231.291764] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4233.718310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4236.015566] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4237.870602] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4240.456713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4240.485805] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4240.542317] Lustre: lustre-OST0001: new disk, initializing [ 4240.544493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4240.570590] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4240.572212] Lustre: Skipped 2 previous similar messages [ 4242.447536] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4243.737861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4243.742824] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4243.762550] LustreError: 307259:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4244.835249] LustreError: 307265:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4244.838283] LustreError: 307265:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [ 4245.858806] LustreError: 307265:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 4245.861439] LustreError: 307265:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 4246.176716] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4247.906573] LustreError: 307259:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4247.909549] LustreError: 307259:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 4248.306698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4249.480525] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92c5a0c74800.ost_server_uuid 50 [ 4250.007069] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92c5a0c74800.ost_server_uuid in FULL state after 0 sec [ 4251.316154] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92c5a0c74800.ost_server_uuid 50 [ 4251.884933] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92c5a0c74800.ost_server_uuid in FULL state after 0 sec [ 4252.003274] LustreError: 307259:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 4252.008733] LustreError: 307259:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 4253.033364] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4265.395716] Lustre: lustre-OST0001: Not available for connect from 192.168.201.44@tcp (stopping) [ 4265.397366] Lustre: Skipped 2 previous similar messages [ 4268.513244] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4268.514822] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.515141] LustreError: Skipped 1 previous similar message [ 4268.518549] Lustre: Skipped 1 previous similar message [ 4269.740077] LustreError: 308136:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4269.741661] LustreError: 308136:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4269.755909] Lustre: server umount lustre-OST0001 complete [ 4270.507452] LustreError: 305911:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4270.511353] LustreError: 305911:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4272.096589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4273.632593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4273.634362] LustreError: 305912:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.635583] Lustre: Skipped 2 previous similar messages [ 4273.646163] LustreError: 305912:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4277.610826] Lustre: server umount lustre-OST0000 complete [ 4282.847811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4282.850664] LustreError: Skipped 1 previous similar message [ 4282.852589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.857951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4282.859448] Lustre: Skipped 5 previous similar messages [ 4293.087150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4293.124811] LustreError: 308538:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4293.126424] LustreError: 308538:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4293.151215] Lustre: server umount lustre-MDT0000 complete [ 4294.111959] LustreError: 304314:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.115551] LustreError: 304314:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4294.312372] LustreError: 304293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680098 with bad export cookie 11789726932308161085 [ 4294.316292] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.316868] LustreError: 304293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4294.462704] Lustre: server umount lustre-MDT0001 complete [ 4298.071468] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 4299.300631] Key type lgssc unregistered [ 4299.446445] LNet: 309272:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4299.450608] LNetError: 309272:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4299.462476] LNet: Removed LNI 192.168.201.144@tcp [ 4299.834764] Key type .llcrypt unregistered [ 4299.835834] Key type ._llcrypt unregistered [ 4309.231967] Key type ._llcrypt registered [ 4309.232870] Key type .llcrypt registered [ 4309.273381] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 4309.768807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4309.794886] alg: No test for adler32 (adler32-zlib) [ 4310.644740] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4310.723095] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 4312.295171] Key type lgssc registered [ 4312.813439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4315.312336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4315.318094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.468786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4317.743572] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4319.825036] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4322.394613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4322.569441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4323.991329] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4327.302622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4327.463500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.207574] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4330.860102] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4332.578899] LustreError: 312202:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.594076] LustreError: 312202:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4333.867452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4333.984447] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4336.010863] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4337.789612] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4341.237294] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4341.240981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4344.794352] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4344.886098] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4426.208284] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4426.211801] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.217440] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4428.261873] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.263038] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4428.263254] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4428.269814] Lustre: Skipped 1 previous similar message [ 4429.934436] LustreError: 314070:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4429.951985] Lustre: server umount lustre-OST0001 complete [ 4433.380038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4433.382152] LustreError: 312946:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.383875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4433.385385] Lustre: Skipped 1 previous similar message [ 4433.392223] LustreError: 312946:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4433.401675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4437.680279] LustreError: 314272:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4437.683983] LustreError: 314272:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4437.760624] Lustre: server umount lustre-OST0000 complete [ 4440.544561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4440.549221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.557439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4440.562284] Lustre: Skipped 1 previous similar message [ 4445.522467] LustreError: 314473:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4445.525311] LustreError: 314473:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4445.561092] Lustre: server umount lustre-MDT0000 complete [ 4447.255928] LustreError: 310629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680251 with bad export cookie 1985690622350857516 [ 4447.262903] LustreError: 310629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4447.263337] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4447.474714] Lustre: server umount lustre-MDT0001 complete [ 4451.352806] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 4452.341746] Key type lgssc unregistered [ 4452.464569] LNet: 315210:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4452.467052] LNetError: 315210:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4452.478341] LNet: Removed LNI 192.168.201.144@tcp [ 4452.785091] Key type .llcrypt unregistered [ 4452.785976] Key type ._llcrypt unregistered [ 4472.019146] Key type ._llcrypt registered [ 4472.020259] Key type .llcrypt registered [ 4472.062529] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 4472.447891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4472.460670] alg: No test for adler32 (adler32-zlib) [ 4473.306167] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4473.387348] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 4474.967202] Key type lgssc registered [ 4475.300090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4478.656591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4478.665589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.842812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.225385] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4484.309760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4484.435908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4485.945449] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4487.025051] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4489.511205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4489.685629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.725856] LustreError: 318917:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.730765] LustreError: 318917:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4490.733383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4491.823856] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4493.795632] LustreError: 318916:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.567604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4494.647449] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4495.178630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 4496.813825] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4501.018636] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4506.782051] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4515.298577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.299594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4515.300734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4515.301408] Lustre: Skipped 2 previous similar messages [ 4520.416624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4520.419243] Lustre: Skipped 5 previous similar messages [ 4521.347620] LustreError: 320866:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4521.376977] Lustre: server umount lustre-MDT0000 complete [ 4522.662514] LustreError: 318177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680327 with bad export cookie 13437069645137587571 [ 4522.667132] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4522.667374] LustreError: 318177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4522.731921] LustreError: 321069:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4522.733782] LustreError: 321069:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4522.807614] Lustre: server umount lustre-MDT0001 complete [ 4533.689127] LustreError: 321270:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4533.692785] LustreError: 321270:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4533.715145] Lustre: server umount lustre-OST0000 complete [ 4545.136257] LustreError: 321472:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4545.137932] LustreError: 321472:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4545.201906] Lustre: server umount lustre-OST0001 complete [ 4550.504396] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 4553.927850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4554.089554] LustreError: 322648:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.098986] LustreError: 322648:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4554.110074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.351670] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4558.146765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4559.595660] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4560.527594] Lustre: 323757:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4562.578488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4562.697826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.700175] Lustre: Skipped 1 previous similar message [ 4563.746820] LustreError: 324112:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.750495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4564.522239] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4567.323133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4568.437994] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 4569.461268] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4572.655128] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4574.211153] Lustre: 325603:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4583.909447] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.911751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4583.912227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4583.912237] Lustre: Skipped 1 previous similar message [ 4583.912577] Lustre: Skipped 2 previous similar messages [ 4586.172959] LustreError: 325709:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4586.174707] LustreError: 325709:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4586.211220] Lustre: server umount lustre-MDT0000 complete [ 4587.628669] LustreError: 325623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680391 with bad export cookie 13437069645137589230 [ 4587.634835] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4587.635825] LustreError: 325623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4587.790114] Lustre: server umount lustre-MDT0001 complete [ 4598.714210] LustreError: 326111:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4598.718886] LustreError: 326111:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4598.745989] Lustre: server umount lustre-OST0000 complete [ 4610.435224] Lustre: server umount lustre-OST0001 complete [ 4611.592348] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 4612.073692] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 4612.570538] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 18:46:56 (1756680416) [ 4619.961981] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 4623.077839] loop0: detected capacity change from 0 to 1099511627776 [ 4623.244568] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 4623.247891] *** VALIDATE xfs *** [ 4623.251537] XFS (loop0): Mounting V5 Filesystem [ 4623.259368] XFS (loop0): Ending clean mount [ 4623.689714] loop1: detected capacity change from 0 to 18691697672192 [ 4637.560144] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4643.103590] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 4646.487105] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 4649.686299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4651.999210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4653.947174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4655.974092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.353406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.375625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.449145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4658.461089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4658.495278] Lustre: lustre-MDT0000: new disk, initializing [ 4658.523678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4658.525237] Lustre: Skipped 1 previous similar message [ 4658.530825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4659.750190] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4663.847845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4663.875270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4663.904633] Lustre: 330680:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4663.924732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4663.926083] Lustre: Skipped 1 previous similar message [ 4663.962234] Lustre: lustre-MDT0001: new disk, initializing [ 4664.005527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4664.007987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4665.187243] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4668.193795] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4669.389648] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4671.703984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4671.724049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4671.812816] Lustre: lustre-OST0000: new disk, initializing [ 4671.814403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4673.023811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4673.027595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4673.067695] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4673.628515] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4676.797926] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4678.142477] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4678.213187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4679.455647] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4679.529069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4682.221080] LustreError: 333619:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4682.222631] LustreError: 333619:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4682.278710] Lustre: server umount lustre-OST0000 complete [ 4684.768060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4684.770080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.773312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4684.774683] Lustre: Skipped 3 previous similar messages [ 4688.354268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4693.472952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4693.474608] Lustre: Skipped 3 previous similar messages [ 4693.496764] Lustre: server umount lustre-MDT0000 complete [ 4694.693346] LustreError: 330671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680499 with bad export cookie 13437069645137590784 [ 4694.697448] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4694.699732] LustreError: 330671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4705.327541] Lustre: server umount lustre-MDT0001 complete [ 4708.057379] XFS (loop0): Unmounting Filesystem [ 4709.596495] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 18:48:33 (1756680513) [ 4712.141531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4712.323376] LustreError: 334840:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.350113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4712.352372] Lustre: Skipped 2 previous similar messages [ 4713.756667] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4716.585878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4718.267605] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4720.046243] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4721.541147] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4724.210191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4726.691303] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4728.884727] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4743.139377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.139553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4743.146596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4743.149167] LustreError: Skipped 1 previous similar message [ 4743.151802] Lustre: Skipped 1 previous similar message [ 4748.977116] LustreError: 337203:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4748.981044] LustreError: 337203:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4749.075874] Lustre: server umount lustre-OST0000 complete [ 4752.864628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4752.868792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4752.874977] Lustre: Skipped 1 previous similar message [ 4756.567704] Lustre: server umount lustre-MDT0000 complete [ 4757.835839] LustreError: 334819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680562 with bad export cookie 13437069645137591666 [ 4757.841245] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4757.845605] LustreError: 334819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4758.002349] Lustre: server umount lustre-MDT0001 complete [ 4762.014601] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 4763.260748] Key type lgssc unregistered [ 4763.385355] LNet: 338140:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4763.388988] LNetError: 338140:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4763.395329] LNet: Removed LNI 192.168.201.144@tcp [ 4763.710176] Key type .llcrypt unregistered [ 4763.711325] Key type ._llcrypt unregistered [ 4774.687081] Key type ._llcrypt registered [ 4774.687990] Key type .llcrypt registered [ 4774.723344] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 4775.042466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4775.074734] alg: No test for adler32 (adler32-zlib) [ 4775.994604] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4776.126918] LNet: Added LNI 192.168.201.144@tcp [8/256/0/180] [ 4777.727276] Key type lgssc registered [ 4778.210167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4781.166989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4781.173801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.357853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4784.113810] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4787.389322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4787.530578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4789.005244] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4791.064194] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4792.870217] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4795.638633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4795.789567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4798.118405] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4799.773430] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4813.285028] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.291268] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4813.291814] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4813.298167] Lustre: Skipped 1 previous similar message [ 4815.286873] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.44@tcp (stopping) [ 4818.401518] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4818.688237] LustreError: 341978:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4818.712722] Lustre: server umount lustre-MDT0001 complete [ 4819.967136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4820.396303] LustreError: 339658:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4820.403108] LustreError: 339658:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4822.750479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4822.785051] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4822.786978] Lustre: Found index 1 for lustre-MDT0001, updating log [ 4822.800577] Lustre: 339653:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4822.922195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4824.213354] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 4828.132764] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4831.719534] LustreError: 343156:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 4831.723246] LustreError: 343156:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 4831.727597] LustreError: 339656:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 4831.748553] LustreError: 343159:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.201.144@tcp: cfg command failed: rc = -17 [ 4831.751060] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 4831.751060] [ 5127.600520] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.44@tcp (stopping) [ 5127.603082] Lustre: Skipped 1 previous similar message [ 5130.209936] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5130.213356] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5130.217624] Lustre: Skipped 1 previous similar message [ 5132.685447] LustreError: 354297:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5132.688946] LustreError: 354297:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5132.715257] LustreError: 339658:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.730815] Lustre: server umount lustre-MDT0001 complete [ 5133.809674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5135.330096] LustreError: 339664:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.343880] LustreError: 339664:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5136.962764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5137.016284] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5137.021434] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5137.048706] Lustre: 339654:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5137.248382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.840136] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 5142.501487] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5146.599584] LustreError: 355513:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5146.601946] LustreError: 355513:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5146.604410] LustreError: 339656:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5146.606639] LustreError: 339656:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5146.626486] LustreError: 355521:0:(obd_config.c:688:class_setup()) Device 8 already setup (type osp) [ 5146.628574] LustreError: 355521:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.201.144@tcp: cfg command failed: rc = -17 [ 5146.630967] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.144@tcp [ 5146.630967] [ 5444.578052] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.578544] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5444.579060] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5444.579064] Lustre: Skipped 2 previous similar messages [ 5444.583229] Lustre: Skipped 2 previous similar messages [ 5446.713284] LustreError: 366962:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5446.714807] LustreError: 366962:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5446.736356] Lustre: server umount lustre-MDT0001 complete [ 5447.596961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5449.696576] LustreError: 339663:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.700100] LustreError: 339663:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5449.944249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5449.973230] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5449.974721] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5449.986276] Lustre: 339654:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5450.097361] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 5451.341557] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 5455.333099] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5458.918943] LustreError: 368163:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 5458.921352] LustreError: 368163:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 5458.923738] LustreError: 339656:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 5458.925394] LustreError: 339656:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [ 5458.944069] LustreError: 368171:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [ 5458.948539] LustreError: 368171:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.201.144@tcp: cfg command failed: rc = -114 [ 5458.950913] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [ 5458.950913] [ 5767.651758] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.653365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5767.657198] Lustre: Skipped 1 previous similar message [ 5767.657930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5767.658960] Lustre: Skipped 4 previous similar messages [ 5771.385835] LustreError: 379906:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5771.387395] LustreError: 379906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5771.417666] Lustre: server umount lustre-MDT0000 complete [ 5772.554651] LustreError: 339645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681576 with bad export cookie 4447911578269357282 [ 5772.558660] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.559950] LustreError: 339645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5772.767976] LustreError: 340880:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5772.768171] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.774572] Lustre: Skipped 1 previous similar message [ 5778.688541] LustreError: 380107:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5778.691254] LustreError: 380107:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5778.794001] Lustre: server umount lustre-MDT0001 complete [ 5780.274584] Lustre: server umount lustre-OST0000 complete [ 5783.010780] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 5786.045753] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 5789.214541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5791.314574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5793.180319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5795.239498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5797.363819] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 19:06:41 (1756681601) [ 5799.382451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5799.406786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5799.480311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5799.492065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5799.525547] Lustre: lustre-MDT0000: new disk, initializing [ 5799.547408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.553229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5800.738210] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 5804.691623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5804.715070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5804.743552] Lustre: 382837:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5804.763223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5804.764617] Lustre: Skipped 1 previous similar message [ 5804.797138] Lustre: lustre-MDT0001: new disk, initializing [ 5804.828058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5804.838202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5804.842199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5806.006917] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 5808.823589] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5809.866826] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5811.839652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5811.861780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5811.950959] Lustre: lustre-OST0000: new disk, initializing [ 5811.953051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5811.976653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5813.582899] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_default_debug -1 all [ 5815.990397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5815.993851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5816.028940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5816.434521] Lustre: DEBUG MARKER: oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5820.383772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5820.385727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5820.388993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5820.391165] Lustre: Skipped 2 previous similar messages [ 5825.018394] LustreError: 385288:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5825.021871] LustreError: 385288:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5825.052859] Lustre: server umount lustre-MDT0000 complete [ 5826.163171] LustreError: 383619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681630 with bad export cookie 4447911578269359235 [ 5826.165748] LustreError: 383619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5826.166452] LustreError: MGC192.168.201.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5826.308626] Lustre: server umount lustre-MDT0001 complete [ 5837.742529] Lustre: server umount lustre-OST0000 complete [ 5845.893389] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing set_hostid [ 5848.816395] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing load_modules_local [ 5852.214949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5854.782737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5856.632642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5858.520648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5859.009500] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5726 sec ========== 19:07:42 (1756681662) [ 5859.483104] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:07:43 (1756681663) === [ 5860.422801] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:07:44 (1756681664) === [ 5872.286402] Lustre: DEBUG MARKER: oleg144-server.virtnet: executing unload_modules_local [ 5873.255321] Key type lgssc unregistered [ 5873.392293] LNet: 390791:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5873.394339] LNetError: 390791:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5873.402289] LNet: Removed LNI 192.168.201.144@tcp [ 5873.687289] Key type .llcrypt unregistered [ 5873.688223] Key type ._llcrypt unregistered