[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 686327260 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002009] Switched APIC routing to physical x2apic. [ 0.003024] kvm-guest: setup PV IPIs [ 0.006693] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009129] LSM: Security Framework initializing [ 0.011059] Yama: becoming mindful. [ 0.012051] SELinux: Initializing. [ 0.013076] *** VALIDATE selinux *** [ 0.024570] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030931] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034020] *** VALIDATE tmpfs *** [ 0.035413] *** VALIDATE proc *** [ 0.037150] *** VALIDATE cgroup *** [ 0.039010] *** VALIDATE cgroup2 *** [ 0.040000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.043005] Speculative Store Bypass: Vulnerable [ 0.047306] debug: unmapping init [mem 0xffffffffbcc59000-0xffffffffbcc60fff] [ 0.051000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052375] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053040] ... version: 2 [ 0.054018] ... bit width: 48 [ 0.055007] ... generic registers: 4 [ 0.056008] ... value mask: 0000ffffffffffff [ 0.057018] ... max period: 00007fffffffffff [ 0.058011] ... fixed-purpose events: 3 [ 0.059020] ... event mask: 000000070000000f [ 0.060335] rcu: Hierarchical SRCU implementation. [ 0.062345] smp: Bringing up secondary CPUs ... [ 0.063638] x86: Booting SMP configuration: [ 0.064030] .... node #0, CPUs: #1 #2 #3 [ 0.080013] smp: Brought up 1 node, 4 CPUs [ 0.082021] smpboot: Max logical packages: 1 [ 0.083011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147028] node 0 deferred pages initialised in 53ms [ 0.153169] devtmpfs: initialized [ 0.154000] x86/mm: Memory block size: 128MB [ 0.160276] gcov: version magic: 0x41383552 [ 0.163577] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165276] pinctrl core: initialized pinctrl subsystem [ 0.166161] [ 0.167012] ************************************************************* [ 0.168013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169010] ** ** [ 0.170009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171020] ** ** [ 0.172014] ** This means that this kernel is built to expose internal ** [ 0.173011] ** IOMMU data structures, which may compromise security on ** [ 0.174017] ** your system. ** [ 0.175013] ** ** [ 0.176014] ** If you see this message and you are not debugging the ** [ 0.177017] ** kernel, report this immediately to your vendor! ** [ 0.178015] ** ** [ 0.179013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180019] ************************************************************* [ 0.182101] NET: Registered protocol family 16 [ 0.183423] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.184060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185115] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186666] cpuidle: using governor menu [ 0.187756] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189000] PCI: Using configuration type 1 for base access [ 0.189000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211096] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.229047] cryptd: max_cpu_qlen set to 1000 [ 0.234619] ACPI: Added _OSI(Module Device) [ 0.235014] ACPI: Added _OSI(Processor Device) [ 0.238013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248423] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256329] ACPI: Interpreter enabled [ 0.257063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.258014] ACPI: Using IOAPIC for interrupt routing [ 0.259118] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260447] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271454] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.274078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276369] acpiphp: Slot [2] registered [ 0.277165] acpiphp: Slot [5] registered [ 0.278219] acpiphp: Slot [6] registered [ 0.279122] acpiphp: Slot [7] registered [ 0.280126] acpiphp: Slot [8] registered [ 0.281091] acpiphp: Slot [9] registered [ 0.282098] acpiphp: Slot [10] registered [ 0.283185] acpiphp: Slot [3] registered [ 0.284098] acpiphp: Slot [4] registered [ 0.285146] acpiphp: Slot [11] registered [ 0.286126] acpiphp: Slot [12] registered [ 0.287106] acpiphp: Slot [13] registered [ 0.288086] acpiphp: Slot [14] registered [ 0.289085] acpiphp: Slot [15] registered [ 0.290068] acpiphp: Slot [16] registered [ 0.291092] acpiphp: Slot [17] registered [ 0.292097] acpiphp: Slot [18] registered [ 0.293100] acpiphp: Slot [19] registered [ 0.294212] acpiphp: Slot [20] registered [ 0.295106] acpiphp: Slot [21] registered [ 0.296141] acpiphp: Slot [22] registered [ 0.297117] acpiphp: Slot [23] registered [ 0.298127] acpiphp: Slot [24] registered [ 0.299109] acpiphp: Slot [25] registered [ 0.300128] acpiphp: Slot [26] registered [ 0.301100] acpiphp: Slot [27] registered [ 0.302181] acpiphp: Slot [28] registered [ 0.303080] acpiphp: Slot [29] registered [ 0.304096] acpiphp: Slot [30] registered [ 0.305091] acpiphp: Slot [31] registered [ 0.306055] PCI host bridge to bus 0000:00 [ 0.307017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.311025] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.312097] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.313174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.315225] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317363] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.326667] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.328017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.329133] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.330809] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331849] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.332050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.334793] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.337015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.346015] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.348840] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.352846] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.355016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.358017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.365025] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.371287] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.374017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.377016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.384017] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.390967] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.393015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.396017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.403015] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.408980] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.411013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.414020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.421016] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.426595] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.429015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.432013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.439014] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.444620] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.447013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.450023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.457015] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.464165] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.465330] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.466410] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.467388] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.468240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.472000] iommu: Default domain type: Passthrough [ 0.479451] SCSI subsystem initialized [ 0.483156] ACPI: bus type USB registered [ 0.494144] usbcore: registered new interface driver usbfs [ 0.506085] usbcore: registered new interface driver hub [ 0.513080] usbcore: registered new device driver usb [ 0.521240] pps_core: LinuxPPS API ver. 1 registered [ 0.528015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.539097] PTP clock support registered [ 0.547191] EDAC MC: Ver: 3.0.0 [ 0.553411] PCI: Using ACPI for IRQ routing [ 0.554000] NetLabel: Initializing [ 0.554000] NetLabel: domain hash size = 128 [ 0.555013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.556078] NetLabel: unlabeled traffic allowed by default [ 0.558334] vgaarb: loaded [ 0.560675] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.562013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.569220] clocksource: Switched to clocksource kvm-clock [ 0.759201] VFS: Disk quotas dquot_6.6.0 [ 0.760688] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.771232] *** VALIDATE ramfs *** [ 0.772363] *** VALIDATE hugetlbfs *** [ 0.790922] pnp: PnP ACPI init [ 0.793905] pnp: PnP ACPI: found 6 devices [ 0.875204] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.883094] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.889758] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.895595] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.901459] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.908170] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.914972] NET: Registered protocol family 2 [ 0.923202] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.931629] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.939688] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.952182] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.958340] TCP: Hash tables configured (established 65536 bind 65536) [ 0.966962] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.975499] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983648] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991344] NET: Registered protocol family 1 [ 0.998892] RPC: Registered named UNIX socket transport module. [ 1.003730] RPC: Registered udp transport module. [ 1.007575] RPC: Registered tcp transport module. [ 1.015247] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.018656] NET: Registered protocol family 44 [ 1.023918] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.030706] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.035898] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.038204] PCI: CLS 0 bytes, default 64 [ 1.039742] Unpacking initramfs... [ 5.243882] debug: unmapping init [mem 0xffff90d47cc54000-0xffff90d47ffbffff] [ 5.251388] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.253712] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.258659] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.768378] Initialise system trusted keyrings [ 5.770171] Key type blacklist registered [ 5.772710] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.782051] zbud: loaded [ 5.785760] *** VALIDATE nfs *** [ 5.787033] *** VALIDATE nfs4 *** [ 5.788609] pstore: using deflate compression [ 5.792301] Platform Keyring initialized [ 6.014563] NET: Registered protocol family 38 [ 6.016284] Key type asymmetric registered [ 6.017958] Asymmetric key parser 'x509' registered [ 6.020714] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.023727] io scheduler mq-deadline registered [ 6.025346] io scheduler kyber registered [ 6.026941] io scheduler bfq registered [ 6.029576] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.032545] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.035671] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.038849] ACPI: Power Button [PWRF] [ 6.147855] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.255996] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.469097] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.568037] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.850779] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.884138] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.914123] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.925337] Non-volatile memory driver v1.3 [ 6.927311] Linux agpgart interface v0.103 [ 6.968712] virtio_blk virtio1: [vda] 133776 512-byte logical blocks (68.5 MB/65.3 MiB) [ 6.974069] vda: detected capacity change from 0 to 68493312 [ 7.000314] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.006569] vdb: detected capacity change from 0 to 1073741824 [ 7.026738] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.030345] vdc: detected capacity change from 0 to 2621440000 [ 7.053243] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.056666] vdd: detected capacity change from 0 to 2621440000 [ 7.076876] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.080154] vde: detected capacity change from 0 to 4294967296 [ 7.100797] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.103946] vdf: detected capacity change from 0 to 4294967296 [ 7.119149] libphy: Fixed MDIO Bus: probed [ 7.124773] usbcore: registered new interface driver usbserial_generic [ 7.127362] usbserial: USB Serial support registered for generic [ 7.129935] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.134597] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.136526] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.139412] mousedev: PS/2 mouse device common for all mice [ 7.143234] rtc_cmos 00:05: RTC can wake from S4 [ 7.147573] rtc_cmos 00:05: registered as rtc0 [ 7.149345] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.151234] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.165905] intel_pstate: CPU model not supported [ 7.172540] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.173457] hid: raw HID events driver (C) Jiri Kosina [ 7.178102] usbcore: registered new interface driver usbhid [ 7.182939] usbhid: USB HID core driver [ 7.184659] drop_monitor: Initializing network drop monitor service [ 7.188721] Initializing XFRM netlink socket [ 7.189740] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.190618] NET: Registered protocol family 10 [ 7.195393] Segment Routing with IPv6 [ 7.196851] NET: Registered protocol family 17 [ 7.199869] mpls_gso: MPLS GSO support [ 7.205857] RAS: Correctable Errors collector initialized. [ 7.208081] AVX version of gcm_enc/dec engaged. [ 7.210101] AES CTR mode by8 optimization enabled [ 7.317132] sched_clock: Marking stable (7317088746, 0)->(11552132472, -4235043726) [ 7.321488] registered taskstats version 1 [ 7.324441] Loading compiled-in X.509 certificates [ 7.327643] zswap: loaded using pool lzo/zbud [ 7.378632] Key type big_key registered [ 7.401838] Key type encrypted registered [ 7.403099] ima: No TPM chip found, activating TPM-bypass! [ 7.405104] ima: Allocated hash algorithm: sha1 [ 7.407870] ima: No architecture policies found [ 7.413878] evm: Initialising EVM extended attributes: [ 7.415814] evm: security.selinux [ 7.417214] evm: security.ima [ 7.418495] evm: security.capability [ 7.419895] evm: HMAC attrs: 0x1 [ 7.422743] rtc_cmos 00:05: setting system clock to 2025-10-23 16:36:33 UTC (1761237393) [ 7.432125] debug: unmapping init [mem 0xffffffffbdc03000-0xffffffffbddfffff] [ 7.446340] debug: unmapping init [mem 0xffffffffbc982000-0xffffffffbcc58fff] [ 7.462084] Write protecting the kernel read-only data: 28672k [ 7.466898] debug: unmapping init [mem 0xffffffffbb003000-0xffffffffbb1fffff] [ 7.471626] debug: unmapping init [mem 0xffffffffbb914000-0xffffffffbb9fffff] [ 7.534473] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.544667] systemd[1]: Detected virtualization kvm. [ 7.546641] systemd[1]: Detected architecture x86-64. [ 7.548633] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.579309] systemd[1]: No hostname configured. [ 7.581643] systemd[1]: Set hostname to . [ 7.584150] random: systemd: uninitialized urandom read (16 bytes read) [ 7.587265] systemd[1]: Initializing machine ID from random generator. [ 7.749231] random: systemd: uninitialized urandom read (16 bytes read) [ 7.751807] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 7.759050] random: systemd: uninitialized urandom read (16 bytes read) [ 7.761326] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 7.768598] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.774870] device-mapper: uevent: version 1.0.3 [ 8.776919] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.740659] virtio_net virtio0 ens2: renamed from eth0 [ 9.779166] random: fast init done [ 9.910412] scsi host0: ata_piix [ 9.943407] scsi host1: ata_piix [ 9.948237] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.950795] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.945599] random: crng init done [ 14.951107] random: 7 urandom warning(s) missed due to ratelimiting [ 15.626092] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.466440] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.440106] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.862811] SELinux: Disabled at runtime. [ 18.936394] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.946403] systemd[1]: Detected virtualization kvm. [ 18.948441] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.754557] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.759885] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.764878] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.771638] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.775560] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.788573] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.806827] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ 19.868728] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 20.878093] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.594687] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.623691] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.947876] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.273768] EDAC sbridge: Ver: 1.1.2 [ 26.386979] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit)[ 26.814637] NFS: Registering the id_resolver key type [ 26.816596] Key type id_resolver registered [ 26.818122] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg627-server login: [ 76.278700] libcfs: loading out-of-tree module taints kernel. [ 76.296082] Key type ._llcrypt registered [ 76.297421] Key type .llcrypt registered [ 76.395330] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 90.533942] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 91.429534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.445098] alg: No test for adler32 (adler32-zlib) [ 92.603684] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 93.090253] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 94.752217] Key type lgssc registered [ 95.639397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.164844] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 109.269086] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.092090] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.511936] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.243216] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 126.367782] hrtimer: interrupt took 6773861 ns [ 138.196457] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 153.197628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.290779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 153.326909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.690396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 154.767377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 154.900960] Lustre: lustre-MDT0000: new disk, initializing [ 155.020061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.045988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.139035] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.084738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.220544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 176.369344] Lustre: 6509:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 176.409403] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 176.418161] Lustre: Skipped 1 previous similar message [ 176.561583] Lustre: lustre-MDT0001: new disk, initializing [ 176.660722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.697844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 176.713336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.306532] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.492164] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 202.067943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.207270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 202.786648] Lustre: lustre-OST0000: new disk, initializing [ 202.789453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.911578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.417772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.449386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.590570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.479675] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.901387] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 229.019660] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 229.122338] Lustre: lustre-OST0001: new disk, initializing [ 229.124826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.212586] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.507965] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.649191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.678074] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.741595] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.830615] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.407484] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 270.451608] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing check_logdir /tmp/testlogs/ [ 275.059150] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing yml_node [ 281.528856] Lustre: DEBUG MARKER: Client: 2.16.59.36 [ 284.648522] Lustre: DEBUG MARKER: MDS: 2.16.59.36 [ 287.989898] Lustre: DEBUG MARKER: OSS: 2.16.59.36 [ 289.550642] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Oct 23 12:41:13 EDT 2025 [ 305.531961] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 306.817963] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 325.089336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 325.092227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.104546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.606552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.612364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.628013] Lustre: Skipped 1 previous similar message [ 325.636610] Lustre: Skipped 2 previous similar messages [ 327.805588] LustreError: 12218:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 327.923716] Lustre: server umount lustre-MDT0000 complete [ 331.402370] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761237717 with bad export cookie 9740784185297379852 [ 331.404775] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 331.425072] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 335.840796] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.840895] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.843484] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.846329] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 335.860973] Lustre: Skipped 2 previous similar messages [ 336.874234] LustreError: 9751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.900276] LustreError: 9751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 337.581660] LustreError: 12420:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 337.589173] LustreError: 12420:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 337.792117] Lustre: server umount lustre-MDT0001 complete [ 341.214095] LustreError: 12624:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 341.246710] LustreError: 12624:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 341.318480] Lustre: server umount lustre-OST0000 complete [ 344.868305] LustreError: 12825:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 344.981216] LustreError: 12825:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 345.315494] Lustre: server umount lustre-OST0001 complete [ 351.324739] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 359.334490] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 368.181643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 374.300921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 379.435378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 385.055972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 391.379542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 391.422041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.575712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 391.606041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 391.652873] Lustre: lustre-MDT0000: new disk, initializing [ 391.756120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.798203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 395.045483] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 404.205138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 404.269751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 404.368521] Lustre: 15010:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 404.465921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 404.468473] Lustre: Skipped 1 previous similar message [ 404.509567] Lustre: lustre-MDT0001: new disk, initializing [ 404.663437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.698594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.703418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.815794] Lustre: 15809:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 404.832037] Lustre: 15809:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 404.848783] Lustre: 15809:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 404.863093] Lustre: 15809:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 404.878293] Lustre: 15809:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 404.899399] Lustre: 15809:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 408.339097] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 415.044483] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 418.424243] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 424.772803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 424.860078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 425.124825] Lustre: lustre-OST0000: new disk, initializing [ 425.131444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 425.233788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.531240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.549529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.665576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 429.989952] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 436.644894] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 440.443844] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 440.957555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 444.215815] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 444.431252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 446.947135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 446.949741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.971578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 446.977613] Lustre: Skipped 3 previous similar messages [ 452.355133] LustreError: 17952:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 452.357772] LustreError: 17952:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 452.492378] Lustre: server umount lustre-OST0000 complete [ 456.162263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 456.173673] LustreError: Skipped 1 previous similar message [ 456.182867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.196106] Lustre: Skipped 1 previous similar message [ 456.211753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 456.216106] Lustre: Skipped 2 previous similar messages [ 462.043159] LustreError: 18153:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 462.049903] LustreError: 18153:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 462.156794] Lustre: server umount lustre-MDT0000 complete [ 462.817533] LustreError: 15018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.841366] LustreError: 15018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 465.219293] LustreError: 15003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761237851 with bad export cookie 9740784185297381427 [ 465.228340] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.558709] Lustre: server umount lustre-MDT0001 complete [ 469.481963] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 12:44:13 (1761237853) [ 475.700069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.166191] LustreError: 18881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.251705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.470537] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 481.256754] LustreError: 18881:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.934975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 490.143665] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 495.003700] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 497.903251] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 503.745865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 508.513225] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 513.016690] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 527.200606] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 533.473202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 533.483108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.494147] Lustre: Skipped 1 previous similar message [ 533.498395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 533.501452] Lustre: Skipped 2 previous similar messages [ 538.835073] LustreError: 21206:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 538.843409] LustreError: 21206:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 539.019664] Lustre: server umount lustre-MDT0000 complete [ 541.987640] LustreError: 18862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761237928 with bad export cookie 9740784185297382344 [ 541.988838] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 542.022221] LustreError: 18862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 542.180159] LustreError: 18875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.187041] LustreError: 18875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 542.406941] Lustre: server umount lustre-MDT0001 complete [ 552.066495] Lustre: server umount lustre-OST0000 complete [ 570.250374] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 578.768317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.278514] LustreError: 22934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.358466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.362547] Lustre: Skipped 2 previous similar messages [ 582.694071] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 589.790663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 593.567770] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 596.257496] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 602.024930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 607.760176] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 615.912321] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 616.027134] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 616.242066] Lustre: lustre-OST0001: new disk, initializing [ 616.252899] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 617.904020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 617.910064] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 618.021821] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 622.291742] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 629.868247] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 633.012510] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 648.677997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 648.683921] Lustre: Skipped 4 previous similar messages [ 648.684260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 648.705794] Lustre: Skipped 6 previous similar messages [ 653.684367] LustreError: 26419:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 653.688404] LustreError: 26419:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 653.759695] Lustre: server umount lustre-MDT0000 complete [ 653.794766] LustreError: 26420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.802806] LustreError: 26420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 657.114658] LustreError: 25210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238043 with bad export cookie 9740784185297383198 [ 657.119070] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.137552] LustreError: 25210:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 657.567104] Lustre: server umount lustre-MDT0001 complete [ 670.961792] Lustre: server umount lustre-OST0000 complete [ 683.316890] Lustre: server umount lustre-OST0001 complete [ 695.094836] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 704.165584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.937567] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 714.330785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 714.672346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 714.675472] Lustre: Skipped 4 previous similar messages [ 717.727108] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 720.115142] Lustre: 29317:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 725.357799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 729.839626] LustreError: 29672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 729.861055] LustreError: 29672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 730.714975] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 739.394084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 744.709443] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 751.782831] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 754.815304] Lustre: 31175:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 765.414112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 765.417513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 765.422916] Lustre: Skipped 1 previous similar message [ 771.302930] LustreError: 31376:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 771.305341] LustreError: 31376:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 771.364866] Lustre: server umount lustre-MDT0000 complete [ 774.182415] LustreError: 28180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238160 with bad export cookie 9740784185297384493 [ 774.187783] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 774.559147] Lustre: server umount lustre-MDT0001 complete [ 788.502132] Lustre: server umount lustre-OST0000 complete [ 791.009197] Lustre: 3655:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761238161/real 1761238161] req@ffff90d4f1a4ca80 x1846791351078144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761238177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 791.881811] Lustre: server umount lustre-OST0001 complete [ 799.509809] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 12:49:43 (1761238183) [ 821.285294] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 829.843606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.091989] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 841.942433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 846.630458] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 849.281385] Lustre: 35080:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 854.700351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 859.529959] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 861.172158] LustreError: 35434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 861.208361] LustreError: 35434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 866.902840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 872.456647] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 879.173475] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 882.200603] Lustre: 36936:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 893.410014] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 893.420573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.426111] Lustre: Skipped 3 previous similar messages [ 893.430590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 893.433777] Lustre: Skipped 7 previous similar messages [ 897.174078] Lustre: server umount lustre-MDT0000 complete [ 900.189580] LustreError: 33940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238286 with bad export cookie 9740784185297385599 [ 900.199737] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 900.220659] LustreError: 33940:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 900.391098] LustreError: 37288:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 900.394330] LustreError: 37288:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 900.636316] Lustre: server umount lustre-MDT0001 complete [ 914.520374] Lustre: server umount lustre-OST0000 complete [ 928.412132] Lustre: server umount lustre-OST0001 complete [ 935.208963] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 12:51:59 (1761238319) [ 949.609851] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 958.123253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.941814] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 969.644152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 974.200686] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 976.733876] Lustre: 40183:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 982.444230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 982.731374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.742891] Lustre: Skipped 8 previous similar messages [ 987.301655] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 995.056249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1000.267920] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1007.003847] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1009.932170] Lustre: 42036:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1015.815269] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1015.820200] Lustre: Skipped 1 previous similar message [ 1023.235113] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.27@tcp (stopping) [ 1023.239795] Lustre: Skipped 8 previous similar messages [ 1024.383853] Lustre: server umount lustre-MDT0000 complete [ 1027.473844] LustreError: 39044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238413 with bad export cookie 9740784185297386705 [ 1027.481146] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.491035] LustreError: 39044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1031.141740] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1031.155130] Lustre: Skipped 8 previous similar messages [ 1033.933507] Lustre: server umount lustre-MDT0001 complete [ 1039.903989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.640661] LustreError: 3651:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff90d4d75e9c00 x1846791351183232/t0(0) o250->MGC192.168.206.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1056.786904] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1062.250503] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1064.295487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1068.027843] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1070.055640] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1070.065331] Lustre: Skipped 1 previous similar message [ 1072.167672] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1073.645755] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1076.361580] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1077.695850] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1085.410579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1087.515197] Lustre: server umount lustre-MDT0000 complete [ 1091.163737] LustreError: 39044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238477 with bad export cookie 9740784185297387895 [ 1091.170400] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1091.729786] Lustre: server umount lustre-MDT0001 complete [ 1105.992831] Lustre: server umount lustre-OST0000 complete [ 1119.661717] Lustre: server umount lustre-OST0001 complete [ 1127.268289] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 12:55:11 (1761238511) [ 1141.690385] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1150.083199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.543103] LustreError: 46413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.562463] LustreError: 46413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 36 previous similar messages [ 1153.768731] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1161.289709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1165.333373] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1168.077318] Lustre: 47533:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1173.623551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.126468] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1185.929681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1190.723808] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1197.094111] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1205.519216] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1218.025669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1221.021550] LustreError: 49610:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1221.024942] LustreError: 49610:0:(obd_class.h:479:obd_check_dev()) Skipped 59 previous similar messages [ 1221.095480] Lustre: server umount lustre-MDT0000 complete [ 1224.126050] LustreError: 46394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761238610 with bad export cookie 9740784185297388679 [ 1224.130489] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1224.134334] LustreError: 46394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1224.501282] Lustre: server umount lustre-MDT0001 complete [ 1237.575912] Lustre: server umount lustre-OST0000 complete [ 1251.002448] Lustre: server umount lustre-OST0001 complete [ 1260.234734] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 12:57:24 (1761238644) [ 1262.284196] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1263.892202] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 12:57:28 (1761238648) [ 1271.733774] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 1277.769838] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1285.154965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1289.341931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1294.653994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1294.715557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.926174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1295.002448] Lustre: lustre-MDT0000: new disk, initializing [ 1295.103909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1298.390557] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1303.396129] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1307.403214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1307.443341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1307.634781] Lustre: lustre-OST0000: new disk, initializing [ 1307.642055] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1307.646746] Lustre: Skipped 1 previous similar message [ 1309.175715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1309.181232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1309.206389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1311.661908] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1316.891328] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1318.215873] Lustre: DEBUG MARKER: create test files [ 1338.000809] Lustre: server umount lustre-OST0000 complete [ 1351.864181] Lustre: server umount lustre-MDT0000 complete [ 1359.011499] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1361.186533] Key type lgssc unregistered [ 1361.425965] LNet: 54951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.434287] LNetError: 54951:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1361.452687] LNet: Removed LNI 192.168.206.127@tcp [ 1362.150466] Key type .llcrypt unregistered [ 1362.152143] Key type ._llcrypt unregistered [ 1380.665557] Key type ._llcrypt registered [ 1380.667431] Key type .llcrypt registered [ 1380.745677] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1381.471437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.488263] alg: No test for adler32 (adler32-zlib) [ 1382.504454] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 1382.683395] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1384.320224] Key type lgssc registered [ 1385.007948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1390.446643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1390.460987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.844802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.631211] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1398.005974] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1402.329282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1402.552765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1406.268847] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1409.905186] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1416.814496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:96 to 0x240000400:129) [ 1416.882689] Lustre: 57531:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1416.887226] Lustre: 57531:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1416.891312] Lustre: 57531:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1416.895208] Lustre: 57531:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1416.899296] Lustre: 57531:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1416.903710] Lustre: 57531:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1424.136818] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1427.610446] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1432.548318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1432.551929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.561515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.676326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1438.008578] LustreError: 58295:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1438.168787] Lustre: server umount lustre-OST0000 complete [ 1441.066907] LustreError: 58498:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1441.071420] LustreError: 58498:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1441.305206] Lustre: server umount lustre-MDT0000 complete [ 1448.624459] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1450.746486] Key type lgssc unregistered [ 1451.039494] LNet: 59030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.061094] LNetError: 59030:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.136116] LNet: Removed LNI 192.168.206.127@tcp [ 1452.880988] Key type .llcrypt unregistered [ 1452.885974] Key type ._llcrypt unregistered [ 1474.566959] Key type ._llcrypt registered [ 1474.568431] Key type .llcrypt registered [ 1474.718509] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1475.820734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1475.847892] alg: No test for adler32 (adler32-zlib) [ 1476.842469] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 1477.014759] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1478.656188] Key type lgssc registered [ 1479.519799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1485.115587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1485.142124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1486.538084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1489.155994] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1492.453398] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1496.718149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1496.899814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.045190] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1503.133900] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1511.425837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1511.527481] Lustre: 61726:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 515 < left 544, rollback = 9 [ 1511.532443] Lustre: 61726:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 32/128/0, destroy: 0/0/0 [ 1511.538784] Lustre: 61726:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1511.544988] Lustre: 61726:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 2/13/1, punch: 0/0/0, quota 1/3/0 [ 1511.547546] Lustre: 61726:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1511.557491] Lustre: 61726:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1518.595856] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1526.755276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.756124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1526.763267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1530.556220] LustreError: 62441:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1530.693386] Lustre: server umount lustre-OST0000 complete [ 1533.526080] LustreError: 62642:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 1533.529101] LustreError: 62642:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1533.767681] Lustre: server umount lustre-MDT0000 complete [ 1542.289443] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 1544.714472] Key type lgssc unregistered [ 1544.955064] LNet: 63174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.975239] LNetError: 63174:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.986790] LNet: Removed LNI 192.168.206.127@tcp [ 1545.596309] Key type .llcrypt unregistered [ 1545.598991] Key type ._llcrypt unregistered [ 1559.051754] Key type ._llcrypt registered [ 1559.053547] Key type .llcrypt registered [ 1559.170438] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 1569.241172] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1569.922950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.937682] alg: No test for adler32 (adler32-zlib) [ 1570.909279] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 1571.080723] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 1572.704548] Key type lgssc registered [ 1573.586466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1580.177848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1586.744128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1592.206036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1596.967065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1601.525361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1601.566495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1601.577467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1602.742443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1602.766524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1602.816580] Lustre: lustre-MDT0000: new disk, initializing [ 1602.891270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1602.909802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1605.771684] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1613.350780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1613.407197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1613.484392] Lustre: 66279:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1613.535941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1613.538448] Lustre: Skipped 1 previous similar message [ 1613.594916] Lustre: lustre-MDT0001: new disk, initializing [ 1613.619256] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1613.699602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1613.741590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1613.749532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1613.788180] Lustre: 67083:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 1613.802090] Lustre: 67083:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 1613.806783] Lustre: 67083:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 1613.815032] Lustre: 67083:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 1613.821027] Lustre: 67083:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 1613.825368] Lustre: 67083:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 1616.492919] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1622.154665] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1624.676677] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1629.531633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1629.580699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1629.755636] Lustre: lustre-OST0000: new disk, initializing [ 1629.758610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1629.829814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1633.997659] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1638.465834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1638.476583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1638.579642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1639.264116] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1641.774026] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1641.893886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1644.576342] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1644.730375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1649.128748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1649.129086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1649.133240] LustreError: Skipped 1 previous similar message [ 1649.135168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1649.143089] Lustre: Skipped 1 previous similar message [ 1654.245471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1654.249424] Lustre: Skipped 2 previous similar messages [ 1659.369258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1660.384136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1660.444129] LustreError: 69220:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1660.686768] Lustre: server umount lustre-OST0000 complete [ 1664.483691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.486146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.490867] Lustre: Skipped 1 previous similar message [ 1664.503274] Lustre: Skipped 1 previous similar message [ 1674.727902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.731224] Lustre: Skipped 4 previous similar messages [ 1678.304149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1678.460933] LustreError: 69424:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1678.470716] LustreError: 69424:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1678.542026] Lustre: server umount lustre-MDT0000 complete [ 1681.599059] LustreError: 67062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239067 with bad export cookie 6597921146369567641 [ 1681.603697] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1681.614454] LustreError: 67062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1681.747641] LustreError: 69626:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1681.750609] LustreError: 69626:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1681.941570] Lustre: server umount lustre-MDT0001 complete [ 1689.122775] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 13:04:33 (1761239073) [ 1697.498310] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1708.102889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1726.537497] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 1732.791986] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 1739.353782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1743.526441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1747.808559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1752.511432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1758.256740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1758.301669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.588038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1758.627635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1758.720510] Lustre: lustre-MDT0000: new disk, initializing [ 1758.807714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.818904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1761.751958] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1768.680884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1768.747273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1768.835798] Lustre: 73874:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1768.893098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1768.896275] Lustre: Skipped 1 previous similar message [ 1768.941213] Lustre: lustre-MDT0001: new disk, initializing [ 1769.042951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1769.069015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1769.087475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1769.130377] Lustre: 74671:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 1769.143166] Lustre: 74671:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 1769.148161] Lustre: 74671:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 1769.153237] Lustre: 74671:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1769.158824] Lustre: 74671:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 1769.167401] Lustre: 74671:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1769.172615] Lustre: 74671:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 1769.177992] Lustre: 74671:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1769.181496] Lustre: 74671:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 1769.187220] Lustre: 74671:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1769.191381] Lustre: 74671:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 1769.200617] Lustre: 74671:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1771.958692] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1779.503811] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1782.895600] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1789.950819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1789.989642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1790.295363] Lustre: lustre-OST0000: new disk, initializing [ 1790.303631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1790.392416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1795.392281] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1796.177302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1796.185926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1796.333952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1801.954644] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1805.945580] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1806.182355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1810.327913] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1810.594351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1817.063224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1817.063640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.067857] LustreError: Skipped 1 previous similar message [ 1817.073858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.087685] Lustre: Skipped 1 previous similar message [ 1818.576487] LustreError: 76816:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1818.580771] LustreError: 76816:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1818.785694] Lustre: server umount lustre-OST0000 complete [ 1822.178616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.195630] Lustre: Skipped 1 previous similar message [ 1828.006452] LustreError: 77018:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1828.009674] LustreError: 77018:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1828.069425] Lustre: server umount lustre-MDT0000 complete [ 1830.963354] LustreError: 73867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239217 with bad export cookie 6597921146369568558 [ 1830.964071] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.971492] LustreError: 73867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1831.279141] Lustre: server umount lustre-MDT0001 complete [ 1837.761424] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 13:07:02 (1761239222) [ 1842.313615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.677245] LustreError: 77890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1842.689775] LustreError: 77890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1842.740255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.238900] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1847.780033] LustreError: 77890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1851.078752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1853.978116] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1857.977668] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1860.682256] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1866.208442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1871.064971] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1875.808752] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1878.858116] Lustre: 80229:0:(genops.c:1791:obd_export_evict_by_uuid()) MGS: evicting d995b1da-125c-4aa8-9f79-5dbd3a7d1b7b at adminstrative request [ 1882.090860] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1882.097717] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 1892.324730] Lustre: MGS: Client d995b1da-125c-4aa8-9f79-5dbd3a7d1b7b (at 0@lo) reconnecting [ 1892.336121] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x5b908653496572d1 to 0x5b90865349657571 [ 1892.362368] Lustre: MGC192.168.206.127@tcp: Connection restored to 0@lo (at 0@lo) [ 1915.078369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1915.149558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1915.315942] Lustre: lustre-OST0001: new disk, initializing [ 1915.324597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1915.415942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1915.419840] Lustre: Skipped 2 previous similar messages [ 1919.485365] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 1921.090312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1921.097964] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1921.149735] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1924.984912] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 1931.238717] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.244487] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1931.257571] Lustre: Skipped 1 previous similar message [ 1931.267199] Lustre: Skipped 6 previous similar messages [ 1933.058739] LustreError: 81364:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1933.064215] LustreError: 81364:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1933.148681] Lustre: server umount lustre-OST0001 complete [ 1936.357100] LustreError: 80255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.402521] LustreError: 80255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1938.407323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1939.428538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.435820] Lustre: Skipped 1 previous similar message [ 1941.480551] LustreError: 79486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1941.496716] LustreError: 79486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1942.661847] Lustre: server umount lustre-OST0000 complete [ 1952.035320] LustreError: 81768:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1952.037963] LustreError: 81768:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1952.125968] Lustre: server umount lustre-MDT0000 complete [ 1954.745778] LustreError: 77869:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239340 with bad export cookie 6597921146369570161 [ 1954.747869] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1954.763241] LustreError: 77869:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1955.031166] Lustre: server umount lustre-MDT0001 complete [ 1961.156339] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 13:09:05 (1761239345) [ 1962.361865] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 1963.683431] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 13:09:08 (1761239348) [ 1964.955962] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 1966.493422] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 13:09:10 (1761239350) [ 1967.858793] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 1969.062699] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 13:09:13 (1761239353) [ 1970.334112] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 1974.182280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1974.944884] loop: module loaded [ 1974.971075] loop0: detected capacity change from 0 to 4294967296 [ 1975.003893] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 1976.112267] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 1978.366778] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 1981.240783] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 1988.716777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1992.618970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1995.976746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1999.389827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2004.689667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2004.760745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2004.790465] LustreError: 84546:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2004.795210] LustreError: 84546:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2004.801756] LustreError: 84546:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2004.807466] LustreError: 84546:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2018.176511] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 2025.809774] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2035.165544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2040.769302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2046.237390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2052.773899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2058.967864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2059.020775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.181620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2059.213195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2059.280364] Lustre: lustre-MDT0000: new disk, initializing [ 2059.368289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.395657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2062.745258] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2071.663718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2071.720515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2071.780563] Lustre: 87390:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2071.850234] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2071.852952] Lustre: Skipped 8 previous similar messages [ 2072.013236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2072.128038] Lustre: 88190:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 2072.132472] Lustre: 88190:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 2072.135583] Lustre: 88190:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 2072.171363] Lustre: 88190:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2072.184872] Lustre: 88190:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 2072.203130] Lustre: 88190:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2072.207583] Lustre: 88190:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 2072.216903] Lustre: 88190:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2072.228177] Lustre: 88190:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 2072.240052] Lustre: 88190:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2072.251381] Lustre: 88190:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 2072.271665] Lustre: 88190:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2075.120881] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2081.682903] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2085.271880] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2092.397647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2092.467800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2097.988382] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2098.272959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2098.358903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2104.814831] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2108.814532] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2109.023980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2113.014887] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2113.324715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2119.137426] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2119.142173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.158574] Lustre: Skipped 2 previous similar messages [ 2119.161039] LustreError: Skipped 1 previous similar message [ 2121.483045] LustreError: 90325:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2121.486263] LustreError: 90325:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2121.716565] Lustre: server umount lustre-OST0000 complete [ 2128.354794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2131.313129] Lustre: server umount lustre-MDT0000 complete [ 2133.782841] LustreError: 87380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239519 with bad export cookie 6597921146369570756 [ 2133.786776] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2133.795207] LustreError: 87380:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2134.101171] Lustre: server umount lustre-MDT0001 complete [ 2140.177305] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 13:12:04 (1761239524) [ 2145.169932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.447374] LustreError: 91400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.330327] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2154.027664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2157.055257] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2160.521197] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2162.632212] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2167.115257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2171.328657] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2175.050796] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2177.499546] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2177.618611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2179.882862] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2180.027953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2184.155514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2184.206467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2184.295873] Lustre: lustre-OST0001: new disk, initializing [ 2184.299231] Lustre: Skipped 2 previous similar messages [ 2184.301956] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2184.304385] Lustre: Skipped 3 previous similar messages [ 2187.656078] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2192.914809] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2192.948294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2192.956779] Lustre: Skipped 2 previous similar messages [ 2192.967487] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2193.052398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2195.615605] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2195.751744] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2198.422727] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2198.547117] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2212.055751] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2232.786841] Lustre: Failing over lustre-MDT0000 [ 2233.111767] LustreError: 96307:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2233.119933] LustreError: 96307:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2233.218588] Lustre: server umount lustre-MDT0000 complete [ 2234.343558] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.345149] LustreError: 91397:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.351047] Lustre: Skipped 6 previous similar messages [ 2234.362109] LustreError: 91397:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2250.192129] Lustre: 64492:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761239620/real 1761239620] req@00000000c5e8f978 x1846792900825856/t0(0) o400->MGC192.168.206.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761239636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2250.213938] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2251.477113] LustreError: 94852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2251.490388] LustreError: 94852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [ 2254.876520] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2254.878852] LDISKFS-fs (dm-0): recovery complete [ 2254.889358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.476307] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5b90865349663ffa [ 2260.493755] Lustre: MGC192.168.206.127@tcp: Connection restored to 0@lo (at 0@lo) [ 2260.801760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 [ 2260.808181] Lustre: Skipped 6 previous similar messages [ 2260.890125] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2261.719190] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2261.725181] Lustre: 95721:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2265.426128] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2266.095458] Lustre: 92113:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2266.102801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2266.110804] Lustre: 92113:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 2266.207448] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2266.529250] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2266.860211] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2266.866726] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2267.193680] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2267.199500] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 1 previous similar message [ 2267.213818] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2267.219672] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 7 previous similar messages [ 2267.865718] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2267.872318] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 2 previous similar messages [ 2268.512554] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2268.519201] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 3 previous similar messages [ 2269.498268] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2269.524530] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 6 previous similar messages [ 2270.171638] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2270.185802] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 2270.832170] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2270.840103] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 6 previous similar messages [ 2273.700147] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2273.716550] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 12 previous similar messages [ 2274.361321] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2274.368890] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2275.016138] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2275.023972] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 12 previous similar messages [ 2281.960966] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2281.968305] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 25 previous similar messages [ 2282.611713] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2282.620229] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 25 previous similar messages [ 2283.256138] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2283.258416] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 25 previous similar messages [ 2297.977190] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2297.988757] Lustre: 97056:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 49 previous similar messages [ 2298.628809] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2298.640561] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 49 previous similar messages [ 2299.288195] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2299.294509] LustreError: 97056:0:(ldlm_lib.c:2679:replay_request_or_update()) Skipped 49 previous similar messages [ 2321.272194] Lustre: 97056:0:(ldlm_lib.c:2375:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2321.275951] Lustre: 97056:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2321.286461] Lustre: 97056:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 19d32b20-b029-4594-90b8-b5172039ba70@192.168.206.27@tcp [ 2321.299049] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2321.302265] Lustre: 97056:0:(ldlm_lib.c:1897:abort_req_replay_queue()) @@@ aborted: req@00000000fbc1271f x1846792889108096/t0(8589934773) o101->19d32b20-b029-4594-90b8-b5172039ba70@192.168.206.27@tcp:633/0 lens 592/0 e 0 to 0 dl 1761239718 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2321.335587] Lustre: lustre-MDT0000: Denying connection for new client 19d32b20-b029-4594-90b8-b5172039ba70 (at 192.168.206.27@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 [ 2321.341583] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2321.378706] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2321.421959] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2321.425060] Lustre: Skipped 2 previous similar messages [ 2321.427225] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2321.521160] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:45 to 0x2c0000401:545) [ 2321.521813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:77 to 0x280000401:577) [ 2337.765564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.779429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2337.783685] Lustre: Skipped 6 previous similar messages [ 2341.439569] Lustre: server umount lustre-OST0000 complete [ 2342.888445] LustreError: 92996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.911586] LustreError: 92996:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 2346.977332] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2350.477608] Lustre: server umount lustre-OST0001 complete [ 2359.655235] Lustre: server umount lustre-MDT0000 complete [ 2362.782488] LustreError: 91381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239748 with bad export cookie 6597921146369622010 [ 2362.785943] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.791967] LustreError: 91381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2362.915407] LustreError: 98324:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2362.918733] LustreError: 98324:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 2363.129956] Lustre: server umount lustre-MDT0001 complete [ 2370.797334] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 13:15:54 (1761239754) [ 2377.686318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2377.695714] Lustre: *** cfs_fail_loc=197, val=0*** [ 2393.248575] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2398.368672] LustreError: 99017:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.127@tcp: failed processing log, type 4: rc = -110 [ 2403.524144] Lustre: *** cfs_fail_loc=197, val=0*** [ 2403.531134] Lustre: Skipped 65 previous similar messages [ 2428.098236] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2431.942242] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2433.426399] Lustre: server umount lustre-OST0000 complete [ 2438.892208] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 13:17:03 (1761239823) [ 2457.323305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.639612] LustreError: 100525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.675845] LustreError: 100525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2461.792979] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2468.030637] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2471.516657] LustreError: 100555:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2471.527955] LustreError: 100555:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2471.533756] LustreError: 100555:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2471.837403] Lustre: server umount lustre-MDT0000 complete [ 2483.627583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2497.428282] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 2505.907522] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2512.499442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2516.703894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2521.679460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2526.248194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2530.529670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2530.567961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.717115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2530.739659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2530.825292] Lustre: lustre-MDT0000: new disk, initializing [ 2530.910109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.917042] Lustre: Skipped 2 previous similar messages [ 2530.931403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2534.010567] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2542.095078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2542.163339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2542.246849] Lustre: 104597:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2542.425695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2542.494297] Lustre: 105393:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 2542.502972] Lustre: 105393:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 2542.507111] Lustre: 105393:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 2542.512251] Lustre: 105393:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2542.519415] Lustre: 105393:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 2542.525546] Lustre: 105393:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2542.532886] Lustre: 105393:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 2542.541352] Lustre: 105393:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2542.545581] Lustre: 105393:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 2542.551309] Lustre: 105393:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2542.559250] Lustre: 105393:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 2542.566874] Lustre: 105393:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 2545.268786] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2550.441606] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2553.149066] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2558.568047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2558.654984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2563.629357] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2568.359825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2569.244248] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2571.924968] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2572.091805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2574.567227] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2574.692552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2578.913909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2578.915575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.923574] LustreError: Skipped 1 previous similar message [ 2578.929037] Lustre: Skipped 5 previous similar messages [ 2590.176491] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2590.405106] Lustre: server umount lustre-OST0000 complete [ 2594.277653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2594.280651] Lustre: Skipped 13 previous similar messages [ 2599.395676] LustreError: 105391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.412793] LustreError: 105391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2599.456747] Lustre: server umount lustre-MDT0000 complete [ 2601.695467] LustreError: 107736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761239987 with bad export cookie 6597921146369632566 [ 2601.697607] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2601.702228] LustreError: 107736:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2601.918966] Lustre: server umount lustre-MDT0001 complete [ 2607.638920] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 13:19:52 (1761239992) [ 2612.603554] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 2614.423402] Key type lgssc unregistered [ 2614.613338] LNet: 108664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2614.616912] LNetError: 108664:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2615.656561] LNet: Removed LNI 192.168.206.127@tcp [ 2616.343179] Key type .llcrypt unregistered [ 2616.345918] Key type ._llcrypt unregistered [ 2628.959627] Key type ._llcrypt registered [ 2628.962735] Key type .llcrypt registered [ 2629.024100] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 2639.433835] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2640.198805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2640.216859] alg: No test for adler32 (adler32-zlib) [ 2641.188857] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 2641.367522] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 2643.080184] Key type lgssc registered [ 2643.985572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2651.406671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2657.149088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2662.096124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2667.193163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2671.200577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.820581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.859480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2674.867522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.001979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2676.021844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2676.070837] Lustre: lustre-MDT0000: new disk, initializing [ 2676.126038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2676.138196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2678.468301] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2685.573577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2685.644578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2685.846815] Lustre: lustre-OST0000: new disk, initializing [ 2685.850442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2685.853713] Lustre: Skipped 1 previous similar message [ 2685.928533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.653363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2688.659781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2688.682329] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2690.166196] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2698.324722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2698.394407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2698.546395] Lustre: lustre-OST0001: new disk, initializing [ 2698.550173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2698.641403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2703.645265] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2707.584397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 2707.593792] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 2707.622186] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 2712.546385] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.547063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2712.555526] Lustre: Skipped 1 previous similar message [ 2717.367408] LustreError: 114219:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 2717.610185] Lustre: server umount lustre-MDT0000 complete [ 2722.297113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2725.514765] LustreError: 112777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240111 with bad export cookie 13606999039002810113 [ 2725.521323] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.558117] LustreError: 114815:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 2725.566110] LustreError: 114815:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2725.602485] Lustre: server umount lustre-OST0000 complete [ 2727.878961] LustreError: 115016:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2727.881647] LustreError: 115016:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2727.987939] Lustre: server umount lustre-OST0001 complete [ 2734.018593] Lustre: DEBUG MARKER: == conf-sanity test 88: check the default mount options can be overridden ========================================================== 13:21:58 (1761240118) [ 2741.407736] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2747.444066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2748.493685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2753.238849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2754.136045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2759.408836] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 13:22:23 (1761240143) [ 2769.106441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.649864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2772.164262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2773.822877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2774.531600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2776.189039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2788.144312] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 2794.093289] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 2801.195306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2806.354393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2811.107314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2815.639912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2820.770550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2820.840372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.116486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2821.155544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2821.205830] Lustre: lustre-MDT0000: new disk, initializing [ 2821.308291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.326533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2824.323104] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2831.712075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2831.779497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2831.845255] Lustre: 120341:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2831.889263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2831.892058] Lustre: Skipped 1 previous similar message [ 2831.939286] Lustre: lustre-MDT0001: new disk, initializing [ 2832.014127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2832.051099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2832.057109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2832.124430] Lustre: 121141:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 2832.131900] Lustre: 121141:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 2832.138543] Lustre: 121141:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 2832.150161] Lustre: 121141:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 2832.161527] Lustre: 121141:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 2832.170975] Lustre: 121141:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 2834.609284] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2839.525766] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2841.844745] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2845.889583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2845.928234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2846.088913] Lustre: lustre-OST0000: new disk, initializing [ 2846.092622] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2846.161645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2849.866644] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2854.786335] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2855.468597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2855.474788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2855.540051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2857.274714] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2857.441718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2859.950969] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2860.081768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2865.644350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2865.647665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.665856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2865.668616] Lustre: Skipped 1 previous similar message [ 2867.320129] LustreError: 123282:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2867.323199] LustreError: 123282:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2867.443581] Lustre: server umount lustre-OST0000 complete [ 2870.756089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.763056] Lustre: Skipped 1 previous similar message [ 2870.767789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2875.873719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2875.877744] Lustre: Skipped 2 previous similar messages [ 2876.059662] LustreError: 123483:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2876.063234] LustreError: 123483:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2876.134630] Lustre: server umount lustre-MDT0000 complete [ 2878.636620] LustreError: 120332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240264 with bad export cookie 13606999039002811352 [ 2878.636951] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2878.651797] LustreError: 120332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2879.135190] Lustre: server umount lustre-MDT0001 complete [ 2885.425251] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 13:24:29 (1761240269) [ 2891.700620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.350476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2895.160741] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2897.381412] LustreError: 124355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.394171] LustreError: 124355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2901.137831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2904.469297] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2908.995236] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2912.254770] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2918.209783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2922.942810] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 2927.470871] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2941.717275] Lustre: *** cfs_fail_loc=159, val=0*** [ 2958.070946] Lustre: lustre-MDT0000: Client f7f72c50-1ebc-48e8-9218-055b5fe25ca7 (at 192.168.206.27@tcp) reconnecting [ 2959.433487] Lustre: *** cfs_fail_loc=159, val=0*** [ 2959.436535] Lustre: Skipped 5 previous similar messages [ 2975.978576] Lustre: lustre-MDT0000: Client f7f72c50-1ebc-48e8-9218-055b5fe25ca7 (at 192.168.206.27@tcp) reconnecting [ 2982.421193] LustreError: 126870:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2982.424250] LustreError: 126870:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2982.622879] Lustre: server umount lustre-OST0000 complete [ 2988.515197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2988.519400] LustreError: Skipped 1 previous similar message [ 2988.522420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.532767] Lustre: Skipped 1 previous similar message [ 2988.537512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.176680] Lustre: 109985:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761240368/real 1761240368] req@0000000072b63ff7 x1846794022654592/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761240384 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2998.199322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.208967] Lustre: Skipped 1 previous similar message [ 2998.219682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2998.223484] Lustre: Skipped 4 previous similar messages [ 2999.264203] Lustre: 109983:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761240369/real 1761240369] req@0000000026dfd554 x1846794022655360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761240385 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3000.288128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3000.374905] LustreError: 127072:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3000.377940] LustreError: 127072:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3000.458994] Lustre: server umount lustre-MDT0000 complete [ 3003.362309] LustreError: 124355:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.378572] LustreError: 124355:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3003.440421] LustreError: 124337:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240389 with bad export cookie 13606999039002812276 [ 3003.442726] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3003.450343] LustreError: 124337:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3003.852199] Lustre: server umount lustre-MDT0001 complete [ 3012.380334] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3015.088394] Key type lgssc unregistered [ 3015.337525] LNet: 127806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3015.348483] LNetError: 127806:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3015.362413] LNet: Removed LNI 192.168.206.127@tcp [ 3015.968693] Key type .llcrypt unregistered [ 3015.973990] Key type ._llcrypt unregistered [ 3039.153878] Key type ._llcrypt registered [ 3039.155560] Key type .llcrypt registered [ 3039.305616] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3040.541459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3040.620537] alg: No test for adler32 (adler32-zlib) [ 3041.742240] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 3041.964146] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3043.611693] Key type lgssc registered [ 3044.727766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3051.597187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3051.620431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.052455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.835367] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3063.267437] LustreError: 129203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.279314] LustreError: 129203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3063.826410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3064.218734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.685528] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3072.612715] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3075.654760] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3080.746054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3081.033191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3086.002658] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3090.386366] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3098.493698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3108.463817] Lustre: *** cfs_fail_loc=159, val=0*** [ 3124.978406] Lustre: lustre-MDT0000: Client 09637089-99cc-4afe-b038-7d0d907093d9 (at 192.168.206.27@tcp) reconnecting [ 3125.178260] Lustre: 131361:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 3125.192501] Lustre: 131361:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3125.203206] Lustre: 131361:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 3125.211850] Lustre: 131361:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 3125.217479] Lustre: 131361:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3125.225494] Lustre: 131361:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3126.915840] Lustre: *** cfs_fail_loc=159, val=0*** [ 3143.431667] Lustre: lustre-MDT0001: Client 09637089-99cc-4afe-b038-7d0d907093d9 (at 192.168.206.27@tcp) reconnecting [ 3144.710676] Lustre: *** cfs_fail_loc=159, val=0*** [ 3144.724862] Lustre: Skipped 3 previous similar messages [ 3161.350585] Lustre: lustre-MDT0001: Client 09637089-99cc-4afe-b038-7d0d907093d9 (at 192.168.206.27@tcp) reconnecting [ 3164.800781] Lustre: 131933:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3167.412360] Lustre: *** cfs_fail_loc=159, val=0*** [ 3167.417932] Lustre: Skipped 2 previous similar messages [ 3182.863916] Lustre: lustre-MDT0000: Client de38c3ed-7eb3-4e70-8b9d-c1c848b393ed (at 192.168.206.27@tcp) reconnecting [ 3184.479780] Lustre: *** cfs_fail_loc=159, val=0*** [ 3184.488980] Lustre: Skipped 14 previous similar messages [ 3200.797133] Lustre: lustre-MDT0000: Client de38c3ed-7eb3-4e70-8b9d-c1c848b393ed (at 192.168.206.27@tcp) reconnecting [ 3202.510957] Lustre: 132223:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3221.476677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3221.476696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3221.499929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3223.944770] LustreError: 132354:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3224.144930] Lustre: server umount lustre-OST0000 complete [ 3228.130057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3228.133343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3228.154470] Lustre: Skipped 1 previous similar message [ 3228.161417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3228.164233] Lustre: Skipped 1 previous similar message [ 3231.713092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.713489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3231.737630] Lustre: Skipped 1 previous similar message [ 3234.300347] LustreError: 132556:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3234.307662] LustreError: 132556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3236.395176] LustreError: 132556:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3236.411421] LustreError: 132556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3236.482521] Lustre: server umount lustre-MDT0000 complete [ 3236.833618] LustreError: 129203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3236.846447] LustreError: 129203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3240.096802] LustreError: 132243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240626 with bad export cookie 9095742085941852604 [ 3240.104927] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3240.112371] LustreError: 132243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3240.234839] LustreError: 132758:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3240.245459] LustreError: 132758:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3240.510558] Lustre: server umount lustre-MDT0001 complete [ 3252.960700] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3255.559498] Key type lgssc unregistered [ 3255.869770] LNet: 133290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3255.885917] LNetError: 133290:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3255.916903] LNet: Removed LNI 192.168.206.127@tcp [ 3256.878971] Key type .llcrypt unregistered [ 3256.880517] Key type ._llcrypt unregistered [ 3278.714511] Key type ._llcrypt registered [ 3278.716138] Key type .llcrypt registered [ 3278.807318] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3279.786136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3279.843899] alg: No test for adler32 (adler32-zlib) [ 3280.942292] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 3281.163959] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3282.896171] Key type lgssc registered [ 3283.858513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3291.558614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3291.596094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.237379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.181905] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3303.403570] LustreError: 134687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.435969] LustreError: 134687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3305.452384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3305.889600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3309.472402] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3314.450353] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3317.810301] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3324.479024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3324.905550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.048229] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3330.791433] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3335.872903] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3338.229597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3345.307717] Lustre: 136934:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3347.519432] Lustre: 136983:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3348.806316] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3349.510758] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3352.152630] Lustre: 137129:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3352.911615] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3353.963700] Lustre: 137227:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3358.702448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.705307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3358.720311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3363.403108] LustreError: 137336:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3363.812744] Lustre: server umount lustre-OST0000 complete [ 3368.931845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.939951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.939961] Lustre: Skipped 1 previous similar message [ 3368.948722] Lustre: Skipped 2 previous similar messages [ 3374.056317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3374.063318] Lustre: Skipped 2 previous similar messages [ 3378.145686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3378.152048] Lustre: Skipped 1 previous similar message [ 3382.240131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3382.377162] LustreError: 137538:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3382.386754] LustreError: 137538:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3382.540922] Lustre: server umount lustre-MDT0000 complete [ 3383.270544] LustreError: 134687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3383.290977] LustreError: 134687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3386.109664] LustreError: 134667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240772 with bad export cookie 1035454686825747660 [ 3386.124086] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3386.222811] LustreError: 137741:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3386.245627] LustreError: 137741:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3386.586804] Lustre: server umount lustre-MDT0001 complete [ 3397.196380] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3399.894299] Key type lgssc unregistered [ 3400.186265] LNet: 138272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3400.216785] LNetError: 138272:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3400.240699] LNet: Removed LNI 192.168.206.127@tcp [ 3401.095095] Key type .llcrypt unregistered [ 3401.096893] Key type ._llcrypt unregistered [ 3422.130783] Key type ._llcrypt registered [ 3422.132469] Key type .llcrypt registered [ 3422.238833] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3423.114357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3423.130552] alg: No test for adler32 (adler32-zlib) [ 3424.097724] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 3424.283457] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3425.952113] Key type lgssc registered [ 3427.070352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3435.309959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3435.341804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.868540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.985633] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3446.243669] LustreError: 139648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.254587] LustreError: 139648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3449.611750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3449.945031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.252678] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3457.842694] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3461.888824] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3467.707285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3468.098254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.091704] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3474.230536] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 3478.656133] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3481.391842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 3488.148884] Lustre: *** cfs_fail_loc=159, val=0*** [ 3488.159683] Lustre: Skipped 1 previous similar message [ 3504.382568] Lustre: lustre-MDT0000: Client 574a8f15-842f-40ea-9bde-800e3be70027 (at 192.168.206.27@tcp) reconnecting [ 3509.736590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3509.740188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.759567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3510.209353] LustreError: 142025:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3510.241771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3510.247106] LustreError: Skipped 7 previous similar messages [ 3510.249669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.262389] LustreError: 141925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.348366] Lustre: server umount lustre-OST0000 complete [ 3516.386240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3516.392935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.417705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3519.616082] LustreError: 142226:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3519.626040] LustreError: 142226:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3519.781620] Lustre: server umount lustre-MDT0000 complete [ 3522.537511] LustreError: 139643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.568300] LustreError: 139643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3523.469445] LustreError: 139630:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761240909 with bad export cookie 17790307747067339986 [ 3523.481894] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.664747] LustreError: 142427:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3523.667277] LustreError: 142427:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3523.934829] Lustre: server umount lustre-MDT0001 complete [ 3534.780906] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3537.598181] Key type lgssc unregistered [ 3537.860645] LNet: 142960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3537.870296] LNetError: 142960:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3537.892272] LNet: Removed LNI 192.168.206.127@tcp [ 3538.642494] Key type .llcrypt unregistered [ 3538.654195] Key type ._llcrypt unregistered [ 3560.967903] Key type ._llcrypt registered [ 3560.971831] Key type .llcrypt registered [ 3561.059169] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3561.639236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3561.650575] alg: No test for adler32 (adler32-zlib) [ 3562.976782] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 3563.271419] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3565.016161] Key type lgssc registered [ 3566.560117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3575.877802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3575.902793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.575061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.246346] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3587.047739] LustreError: 144361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.068667] LustreError: 144361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3591.234822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3591.608731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3595.311842] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3600.223855] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3604.034934] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3610.630268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3610.896176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3615.877469] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3616.046290] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 3620.612936] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3622.193136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 3642.787376] Lustre: lustre-MDT0001: evicting 036ab228-0d39-49ed-8d9c-cecfeeaf7184 (at 192.168.206.27@tcp) by administrative request [ 3642.793190] Lustre: Skipped 1 previous similar message [ 3672.555183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3672.558373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.569446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.571583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3673.573001] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.574037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.581807] Lustre: Skipped 1 previous similar message [ 3678.697554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.810925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3683.813618] Lustre: Skipped 1 previous similar message [ 3685.344159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3685.396186] LustreError: 147150:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3685.590843] Lustre: server umount lustre-OST0000 complete [ 3694.050913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3694.052244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.128361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3694.147826] Lustre: Skipped 1 previous similar message [ 3695.420579] LustreError: 147352:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3695.425668] LustreError: 147352:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3695.472755] Lustre: server umount lustre-MDT0000 complete [ 3699.170120] LustreError: 144361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.344617] LustreError: 144339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241085 with bad export cookie 17379832320562898494 [ 3699.348638] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3699.354776] LustreError: 144339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3699.464243] LustreError: 147554:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3699.466755] LustreError: 147554:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3699.771811] Lustre: server umount lustre-MDT0001 complete [ 3709.752363] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3712.327972] Key type lgssc unregistered [ 3712.669464] LNet: 148086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3712.677050] LNetError: 148086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3713.705549] LNet: Removed LNI 192.168.206.127@tcp [ 3714.588736] Key type .llcrypt unregistered [ 3714.590257] Key type ._llcrypt unregistered [ 3780.052479] Key type ._llcrypt registered [ 3780.054169] Key type .llcrypt registered [ 3780.177737] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3781.080572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3781.101596] alg: No test for adler32 (adler32-zlib) [ 3782.176515] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 3782.347889] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3783.968188] Key type lgssc registered [ 3784.724787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3789.902413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3789.921428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.270546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3793.869304] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3799.511566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3799.801604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3802.264759] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3805.360400] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3807.868853] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3812.560712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3812.799424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3816.744502] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3816.878060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 3820.617178] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3824.051831] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 3832.290842] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.297463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.306861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3834.342612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3834.343562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3834.346659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.799573] LustreError: 152802:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3836.958845] Lustre: server umount lustre-OST0000 complete [ 3839.461739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.462838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.467357] Lustre: Skipped 1 previous similar message [ 3839.470723] Lustre: Skipped 1 previous similar message [ 3844.577481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3844.580647] Lustre: Skipped 1 previous similar message [ 3845.586855] LustreError: 153003:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3845.592971] LustreError: 153003:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3845.668192] Lustre: server umount lustre-MDT0000 complete [ 3848.329293] LustreError: 150527:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241234 with bad export cookie 2006060409431257219 [ 3848.335171] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3848.435831] LustreError: 153205:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3848.441173] LustreError: 153205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3848.624571] Lustre: server umount lustre-MDT0001 complete [ 3855.433974] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 3857.212706] Key type lgssc unregistered [ 3857.410152] LNet: 153737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3857.416811] LNetError: 153737:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3857.434092] LNet: Removed LNI 192.168.206.127@tcp [ 3858.013722] Key type .llcrypt unregistered [ 3858.016124] Key type ._llcrypt unregistered [ 3874.743454] Key type ._llcrypt registered [ 3874.746124] Key type .llcrypt registered [ 3874.828593] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3875.395378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3875.403147] alg: No test for adler32 (adler32-zlib) [ 3876.370860] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 3876.509192] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 3878.128189] Key type lgssc registered [ 3878.805233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3884.164552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3890.403623] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 13:41:14 (1761241274) [ 3901.448731] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 3907.787213] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 3915.821415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3920.854594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3925.576413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3930.820619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.421406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3936.455938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3936.470470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3937.692476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3937.745176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3937.808118] Lustre: lustre-MDT0000: new disk, initializing [ 3937.911864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.932237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3941.224886] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3950.731403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3950.794686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3950.912452] Lustre: 157938:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3950.996509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3951.004231] Lustre: Skipped 1 previous similar message [ 3951.075295] Lustre: lustre-MDT0001: new disk, initializing [ 3951.208449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.237683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3951.253527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3951.337691] Lustre: 158737:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3951.342093] Lustre: 158737:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3951.350102] Lustre: 158737:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3951.359297] Lustre: 158737:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3951.371103] Lustre: 158737:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3951.381530] Lustre: 158737:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3954.723313] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3961.186912] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3964.003163] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3969.984984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3970.058107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3970.226409] Lustre: lustre-OST0000: new disk, initializing [ 3970.229357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3970.312904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.622547] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 3978.820536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3978.826983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3978.895771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3979.916955] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3989.473872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3989.480028] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.487730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3991.864134] LustreError: 160525:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3992.001175] Lustre: server umount lustre-OST0000 complete [ 3997.154428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3997.167730] LustreError: Skipped 1 previous similar message [ 3997.173900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3997.198475] Lustre: Skipped 1 previous similar message [ 3997.203252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3997.209179] Lustre: Skipped 1 previous similar message [ 3999.715617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.715887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.137619] LustreError: 160726:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4001.142037] LustreError: 160726:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4001.231647] Lustre: server umount lustre-MDT0000 complete [ 4003.964548] LustreError: 157931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761241390 with bad export cookie 9816488336868273943 [ 4003.966819] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4003.974043] LustreError: 157931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4004.070526] LustreError: 160927:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4004.073544] LustreError: 160927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4004.234718] Lustre: server umount lustre-MDT0001 complete [ 4010.404338] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 4012.175929] Key type lgssc unregistered [ 4012.363928] LNet: 161458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4012.369642] LNetError: 161458:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4012.379053] LNet: Removed LNI 192.168.206.127@tcp [ 4012.891522] Key type .llcrypt unregistered [ 4012.893240] Key type ._llcrypt unregistered [ 4028.477881] Key type ._llcrypt registered [ 4028.479147] Key type .llcrypt registered [ 4028.563129] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 4029.220287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4029.245480] alg: No test for adler32 (adler32-zlib) [ 4030.175222] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 4030.306177] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 4031.936166] Key type lgssc registered [ 4032.850941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4038.292663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4038.316844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4039.613334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4041.967902] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4047.688386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4048.040635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4050.678609] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4054.550341] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4056.761352] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4060.864474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4061.078615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.189354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4065.050332] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 4069.077990] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4078.521490] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4078.532925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.543783] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4078.549622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4078.561542] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4079.315074] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4079.333986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.345748] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4079.352598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4079.361900] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4080.137230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4080.835774] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4080.838878] Lustre: Skipped 1 previous similar message [ 4080.847677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.861572] Lustre: Skipped 1 previous similar message [ 4080.865393] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4080.869353] Lustre: Skipped 1 previous similar message [ 4080.877303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4080.888266] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4080.891183] Lustre: Skipped 1 previous similar message [ 4081.536654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4082.236767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4082.915337] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4082.921705] Lustre: Skipped 2 previous similar messages [ 4082.936444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.946641] Lustre: Skipped 2 previous similar messages [ 4082.953791] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4082.961168] Lustre: Skipped 2 previous similar messages [ 4082.968403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4082.974903] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4082.977819] Lustre: Skipped 2 previous similar messages [ 4083.583407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4084.334884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4084.966571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4085.629094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4086.189908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4086.760318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4087.281136] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4087.285247] Lustre: Skipped 6 previous similar messages [ 4087.295544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4087.301710] Lustre: Skipped 6 previous similar messages [ 4087.315336] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4087.317488] Lustre: Skipped 6 previous similar messages [ 4087.322319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4087.327417] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4087.331719] Lustre: Skipped 6 previous similar messages [ 4087.885215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4088.443531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4089.129437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4089.675210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4090.296580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4090.963825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4091.650609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4092.277708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4092.999511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4093.782941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4094.552592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4095.254058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4096.063457] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4096.067773] Lustre: Skipped 12 previous similar messages [ 4096.079139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.087745] Lustre: Skipped 12 previous similar messages [ 4096.093735] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4096.097473] Lustre: Skipped 12 previous similar messages [ 4096.101630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4096.108625] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4096.111315] Lustre: Skipped 12 previous similar messages [ 4097.178804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4098.132236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4099.155311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4099.952137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4100.808492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4101.535958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4102.333532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4103.026390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4103.812649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4104.565487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4105.284607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4106.074609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4106.748566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4107.486433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4108.150828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4108.859119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4109.623543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4110.275882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4110.969976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4111.554292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4112.186730] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4112.197257] Lustre: Skipped 20 previous similar messages [ 4112.230737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.239996] Lustre: Skipped 20 previous similar messages [ 4112.245655] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4112.250756] Lustre: Skipped 20 previous similar messages [ 4112.254342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4112.267793] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4112.270997] Lustre: Skipped 20 previous similar messages [ 4113.028308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4113.796292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4114.424859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4115.089911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4115.920795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4116.721348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4117.497715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4118.381753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4119.418891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4120.193877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4120.919244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4121.675219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4122.507904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4123.247940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4123.929879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4124.677664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4125.393274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4126.078056] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4126.731928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4127.341180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4128.092950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4128.805293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4129.440966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4130.081163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4130.750597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4131.372268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4132.003784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4132.670544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4133.330906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4134.012345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4134.692786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4135.287087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4135.985787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4136.596889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4137.157499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4137.669089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4138.207821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4138.777290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4139.336356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4139.923054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4140.504795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4141.023193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4141.610463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4142.145311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4142.796111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4143.430825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4143.989321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4144.545149] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4144.547786] Lustre: Skipped 47 previous similar messages [ 4144.556946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.563862] Lustre: Skipped 47 previous similar messages [ 4144.567572] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4144.574584] Lustre: Skipped 47 previous similar messages [ 4144.577422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4144.588629] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4144.597397] Lustre: Skipped 47 previous similar messages [ 4145.153477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4145.689899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4146.188825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4146.845657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4147.441303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4147.939734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4148.504814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4148.979733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4149.536909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4150.041528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4150.560135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4151.030637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4151.488889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4151.977552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4152.445909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4152.892211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4153.346198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4153.811379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4154.353141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4154.931160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4155.423776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4155.964034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4156.582276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4157.205703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4157.824901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4158.417679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4158.948063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4159.490321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4160.072960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4160.708623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4161.487522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4162.195156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4162.927818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4163.642862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4164.276784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4165.065419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4165.939834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4166.797540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4167.857468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4168.612324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4169.348083] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4170.126158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4170.954978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4171.738754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4172.513362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4173.178310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4173.785814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4174.444131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4175.180519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4175.848751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4176.631096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4177.313533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4177.973256] LustreError: 162851:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4177.986806] LustreError: 162851:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4177.995787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4178.702693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4179.498732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4180.360034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4181.096497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4181.800870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4182.600576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4183.570741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4184.298705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4185.208833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4186.113779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4186.994476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4187.771066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4188.585882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4189.461410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4190.357168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4191.274509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4192.167763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4193.256668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4194.233046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4195.164402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4196.233030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4197.148775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4197.992584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4198.810333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4199.744581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4200.612749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4201.480863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4202.098933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4202.874764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4203.540449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4204.337561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4205.417171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4206.509527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4207.597812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4208.425802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4209.313041] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4209.317269] Lustre: Skipped 88 previous similar messages [ 4209.328937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4209.334505] Lustre: Skipped 88 previous similar messages [ 4209.338807] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4209.341798] Lustre: Skipped 88 previous similar messages [ 4209.347760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4209.354013] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4209.356368] Lustre: Skipped 88 previous similar messages [ 4210.200072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4211.232942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4212.094206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4212.906190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4213.795890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4214.702291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4215.431876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4216.182690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4216.892750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4217.563269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4218.222086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4218.959892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4219.654437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4220.325661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4220.881242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4221.465612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4222.070801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4222.749076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4223.437594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4224.082497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4224.779980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4225.538667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.148079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4226.828091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4227.545637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4228.374595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4229.052682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4229.690819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4230.453135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4231.189173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4231.975551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4232.795463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4233.582755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4234.294382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4234.878645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4235.457703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4236.037340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4236.699957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4237.388369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4238.249590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4239.185549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4240.015254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4240.843788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4241.634755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4242.381429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4243.176447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4243.801386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4244.351360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4244.904307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4245.452040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4245.997312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4246.754660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4247.359867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4248.015952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4248.666122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4249.395777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4250.085192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4250.768984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4251.425128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4252.175667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4253.103330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4253.922462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4254.657642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4255.594258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4256.527341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4257.603631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4258.620711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4259.525155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4260.295601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4261.127958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4261.793674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4262.500431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4263.279255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4264.137721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4264.891990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4265.600306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4266.226341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4266.923625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4267.626462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4268.339959] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4269.025174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4269.834921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4270.501387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4271.168524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4271.852986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4272.597066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4273.233599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4273.955319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4274.690439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4275.412776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4276.070831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4276.861835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4277.504448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4278.206377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4278.801614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4279.440929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4280.085513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4280.731236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4281.373314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.087649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4282.767425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4283.364548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4283.964179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4284.584301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4285.339924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4286.107812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4286.838080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4287.429735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4288.071282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4288.774217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4289.445098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4290.262183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4291.012422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4291.842201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4292.632031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4293.377553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4294.047668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4294.737271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4295.314295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4295.884435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4296.455416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4297.056528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4297.647729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4298.306487] LustreError: 162851:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4298.315251] LustreError: 162851:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4298.320440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4298.981536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4299.682405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4300.415347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4301.189451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4301.860597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4302.640675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4303.276316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4303.857075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4304.529279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4305.248656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4305.940672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4306.641156] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4307.287261] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4307.911525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4308.591909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4309.194484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4309.796448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4310.518135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4311.176506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4311.872311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4312.553388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4313.230152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4313.992515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4314.713712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4315.435748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4316.136843] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4316.992477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4317.853765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4318.777428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4319.668572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4320.682102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4321.449125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4322.239256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4322.968916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4323.637742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4324.331572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4325.273442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4326.266474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4327.231829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4328.243520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4329.092656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4329.941465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4330.853476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4331.635101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4332.441144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4333.376857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4334.118218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4334.855520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4335.570343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4336.313062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4336.896831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4337.574441] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4337.584064] Lustre: Skipped 175 previous similar messages [ 4337.596466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4337.605663] Lustre: Skipped 175 previous similar messages [ 4337.610947] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4337.614113] Lustre: Skipped 175 previous similar messages [ 4337.621114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4337.629943] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4337.632277] Lustre: Skipped 175 previous similar messages [ 4338.316943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4339.029722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4339.629343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4340.295137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4340.955535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4341.661247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4342.350137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4343.145132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4343.887284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4344.692350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4345.346880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4345.927190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4346.564287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4347.217810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4347.954785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4348.642468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4349.226385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4349.845324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4350.514364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4351.342426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4352.062943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4352.786964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4353.449527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4354.292703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4355.087379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4356.001606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4356.782346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4357.599081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4358.438204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4359.302251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4360.013467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4360.936592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4361.838265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4362.702266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4363.623382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4364.473671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4365.328112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4366.301491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4367.097147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4368.033380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4368.823224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4369.563725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4370.309719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4371.040627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4371.881180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4372.724446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4373.552050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4374.564403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4375.340741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4376.350676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4377.290748] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4378.012752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4378.818171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4379.761041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4380.587786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4381.675557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4382.533172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4383.421544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4384.353710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4385.164668] LustreError: 162851:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4385.176752] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4385.183922] LustreError: 162851:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4385.889341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4386.718492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4387.547160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4388.401403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4389.346627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4389.953363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4390.456484] LustreError: 162851:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -108 [ 4390.462505] LustreError: 162851:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -108 [ 4390.463835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4391.414645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4392.306937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4393.105664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4393.915481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4394.742668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4395.533686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4396.370128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4397.153164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4397.907574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4398.640389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4399.431217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4400.200788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4400.945832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4401.675230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4402.332071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4402.999548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4403.737984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4404.601553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4405.354371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4406.184518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4407.016764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4407.716406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4408.393015] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4409.111573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4409.732470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4410.460070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4411.153508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4411.749742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4412.437689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4413.043237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4413.655599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4414.260782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4414.882517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4415.445339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4416.120201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4416.791362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4417.416224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4418.105311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4418.727844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4419.376896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4420.018901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4420.609369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4421.298956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4421.914775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4422.522836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4423.182995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4423.784618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4424.363650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4424.977082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4425.532841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4426.102701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4426.727645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4427.398946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4428.018073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4428.623493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4429.228356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4429.820841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4430.374568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4431.007075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4431.720806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4432.389546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4433.183715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4433.799354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4434.569217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4435.141894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4435.853063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4436.513306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4437.302737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4438.157586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4438.917939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4439.538409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4440.325677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4441.052883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4441.761966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4442.375196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4443.026553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4443.630619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4444.197063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4444.923120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4445.902733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4446.648481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4447.422885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4448.177958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4448.957025] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4449.659308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4450.357123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4451.032176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4451.789870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4452.525888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4453.270491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4454.012556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4454.735384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4455.471828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4456.322874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4457.095084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4457.928590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4458.769689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4459.679802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4460.535434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4461.462789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4462.216245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4463.071958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4463.955552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4464.769449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4465.783466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4466.658921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4467.479731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4468.308739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4469.143443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4469.972121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4470.755914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4471.604691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4472.502557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4473.276060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4474.036095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4474.908785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4475.777729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4476.607516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4477.348359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4478.167594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4478.887983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4479.715238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4480.439615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4481.397365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4482.190743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4483.037065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4483.893411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4484.690171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4485.518087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4486.367541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4487.266994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4488.149342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4488.992554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4489.897448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4490.632643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4491.365362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4492.011940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4492.909210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4493.691874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4494.468916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4495.289059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4496.123117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4497.052464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4497.780689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4498.592267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4499.431365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4500.211478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4501.000377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4501.751129] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4503.183579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4504.078719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4505.050917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4505.897462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4506.684272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4507.588565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4508.436771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4509.278846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4510.124893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4510.967171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4511.888069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4512.671401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4513.658850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4514.523994] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4515.329670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4516.066149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4516.857684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4517.602765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4518.422247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.154456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4519.884945] LustreError: 162851:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4519.890159] LustreError: 162851:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4519.901470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4520.559799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4521.307971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4522.205332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4522.920769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4523.575677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4524.352101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4525.086097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4525.858779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4526.615125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4527.405593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4528.244407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4528.957522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4529.801462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4530.607606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4531.300267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4532.082872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4532.846108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4533.521879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4534.291721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4535.063380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4535.902669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4536.640038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4537.568636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4538.296681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4539.045930] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4539.843785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4540.791111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4541.470685] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4542.341706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4543.206795] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4543.998620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4544.918066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4545.958061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4546.674212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4547.621379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4548.488763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4549.372493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4550.257074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4551.473120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4552.338284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4553.445665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4554.539599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4555.577110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4556.435886] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4557.324919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4558.145269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4559.223640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4560.240245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4561.303692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4562.468841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4563.674449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4564.713350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4565.890449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4566.847573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4567.843444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4568.739836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4569.855040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4570.869526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4571.755807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4572.801454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4574.031725] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4575.014202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4576.061353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4576.991133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4577.893154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4578.837864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4579.849106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4580.850500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4581.733785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4582.616483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4583.439973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4584.303577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4585.207061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4586.233307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4587.193683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4588.130938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4589.062497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4590.083900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4590.809941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4591.543740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4592.496671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4593.504620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4594.431055] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4594.434440] Lustre: Skipped 319 previous similar messages [ 4594.465468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.473412] Lustre: Skipped 319 previous similar messages [ 4594.479199] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4594.483033] Lustre: Skipped 319 previous similar messages [ 4594.514306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4594.533154] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4594.546521] Lustre: Skipped 319 previous similar messages [ 4595.392699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4596.402110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4597.369692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4598.319586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4599.313416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4600.160021] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4601.015746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4602.000197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4602.848305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4603.815386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4604.648523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4605.373382] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4606.323305] LustreError: 162851:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 4606.337928] LustreError: 162851:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 4606.340756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4607.318087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4608.390123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4609.363115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4610.331232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4611.144333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4611.944833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4612.875642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4613.640831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4614.565451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4615.393714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4616.163679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.078688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4617.900193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4618.803539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4619.559967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4620.369328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4621.191912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4622.055178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4622.807199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4623.695820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4624.711532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4625.786180] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4626.673722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4627.500052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4628.335545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4629.103628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4629.800423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4630.666259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4631.413119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4632.267787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4632.967268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4633.802447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4634.583827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4635.365761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4636.183888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4637.158198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4637.895275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4638.589399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4639.344499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4640.186325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4641.158801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4641.993307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4642.908644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4643.688683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4644.553679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4645.403124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4646.450331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4647.400915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4648.485925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4649.454799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4650.329280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4651.139507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4652.107737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4652.887684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4653.772509] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4654.497933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4655.326077] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4656.112305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4656.863911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4657.618080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4658.564322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4659.357518] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4660.408015] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4661.351946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4662.211699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4662.989455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4663.824263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4664.634136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4665.364534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4666.075211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4666.821965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4667.613831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4668.308641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4669.019961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4669.833862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4670.611307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4671.421981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4672.308285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4673.080269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4673.801429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4674.673837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4675.342879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4676.153581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4676.958141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4677.735560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4678.482924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4679.160294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4679.915206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4680.730593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4681.461869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4682.212582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4682.941641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4683.706393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4684.583379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4685.402405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4686.219979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4687.005638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4687.742245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4688.442687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4689.423659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4690.273410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4691.053229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4692.018379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4692.821956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4693.516037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4694.521755] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4695.480773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4696.472311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4697.230826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4698.220446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4699.196593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4700.108155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4700.959558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4702.016323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4703.221122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4704.045831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4705.009552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4706.052804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4706.953217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4707.832648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4708.635394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4709.545600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4710.373423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4711.318408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4712.060877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4712.872465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4713.646943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4714.442221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4715.243789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4716.169773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4717.036985] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4717.947189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4718.896636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4719.726708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4720.667884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4721.676971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4722.512598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4723.502217] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4724.352412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4725.241579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4726.080613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4726.905668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4727.608373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.270766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4728.998825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4729.767299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4730.623528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4731.466932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4732.370790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4733.230629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4734.155284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4734.917470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4735.825496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4736.663107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4737.485425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4738.211784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4739.032264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4739.783955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4740.621921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4741.375780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4742.379421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4743.308705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4744.046688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4744.718709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4745.492616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4746.459102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4747.430117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4748.237850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4749.119915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4749.977408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4750.956744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4751.802496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4752.626826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4753.310353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4754.117916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4754.903421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4755.758540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4756.545947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4757.301846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4758.193604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.065577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.859871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4760.751593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4761.502642] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4762.405475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4763.206625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4764.095273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4764.959672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4765.797614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4766.625906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4767.474926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4768.218209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4769.190508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.041936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4770.828318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4771.586672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4772.298260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.000252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4773.724037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4774.433179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.151088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4775.795242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4776.476947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4777.172248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.055808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4778.807409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4779.523430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.200336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4780.860033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4781.477496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.172914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4782.829677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4783.607557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4784.470903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.143650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4785.923679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4786.635354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4787.354238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.106939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4788.882736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4789.612368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4790.438519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4791.231532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4792.066526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4792.959616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.786406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4794.450234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.118929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4795.812801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4796.336585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.031721] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.708400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.300402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4798.921918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4799.550579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.241254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4800.925469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4801.718562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4802.339250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4802.945969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4803.763600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4804.512440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4805.212516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.147175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4806.862928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4807.732222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4808.643810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4809.459639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4810.305596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.030856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4811.910248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4812.681246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4813.424703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4814.178057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.137028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4815.988931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4816.788236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4817.478944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4818.326345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4819.193971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4820.250095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4821.110166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4822.185351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.032741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4823.896901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4824.873169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4825.744876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4826.560064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4827.359140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4828.100683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4829.072284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.120769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4830.979410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4831.892215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4832.814490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4833.826709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4834.777113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4835.701453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4836.410724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4837.391051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.161888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4838.846187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4839.574349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4840.385652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4841.241680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4841.996566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4842.896627] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4843.558387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4844.308314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.035713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4845.875672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4846.607031] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4847.253786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.039095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4848.781434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4849.382970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.126144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4850.908850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4851.587092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4852.335133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.102952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.813921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4854.559472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.312475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4855.995974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4856.583412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.500609] LustreError: 164432:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 5249.518824] LustreError: 162851:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 5640.163467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5640.179078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5640.210424] Lustre: Skipped 319 previous similar messages [ 5640.213814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5641.193132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5641.212607] Lustre: Skipped 1 previous similar message [ 5645.753277] LustreError: 216750:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5645.905149] Lustre: server umount lustre-OST0000 complete [ 5651.426654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5651.431860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5651.447636] Lustre: Skipped 1 previous similar message [ 5656.549724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5656.552272] Lustre: Skipped 1 previous similar message [ 5661.668489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5661.676140] Lustre: Skipped 1 previous similar message [ 5663.200828] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5663.291188] LustreError: 216954:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5663.294218] LustreError: 216954:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5663.522129] Lustre: server umount lustre-MDT0000 complete [ 5666.132076] LustreError: 162812:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243052 with bad export cookie 3839193083827073034 [ 5666.140845] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5666.268212] LustreError: 217157:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5666.272764] LustreError: 217157:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5666.479303] Lustre: server umount lustre-MDT0001 complete [ 5673.450694] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 5675.076860] Key type lgssc unregistered [ 5675.272723] LNet: 217688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5675.279707] LNetError: 217688:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5675.293927] LNet: Removed LNI 192.168.206.127@tcp [ 5675.852381] Key type .llcrypt unregistered [ 5675.854726] Key type ._llcrypt unregistered [ 5693.570163] Key type ._llcrypt registered [ 5693.572302] Key type .llcrypt registered [ 5693.673951] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 5694.891940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5694.924366] alg: No test for adler32 (adler32-zlib) [ 5695.872278] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 5696.030720] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 5697.697678] Key type lgssc registered [ 5698.533319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5704.369789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5704.386680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5705.769591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5708.950895] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5715.957543] LustreError: 219064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5715.968641] LustreError: 219064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5716.716237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5717.198916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5720.774033] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5727.241416] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5731.091303] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5737.009793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5737.360226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.893096] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5745.966265] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5751.678877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49666 to 0x280000401:49697) [ 5755.063899] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5756.344537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.352615] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5756.361283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.369935] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5756.371150] LustreError: 219084:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 5757.921493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5757.934436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5757.949199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5762.019375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5762.023229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5763.774037] LustreError: 221466:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5763.905896] Lustre: server umount lustre-OST0000 complete [ 5776.408172] LustreError: 219084:0:(osp_precreate.c:1263:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 5793.924911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5794.135408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5797.594578] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5800.567773] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5802.342914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.351290] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5803.786363] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 5806.735417] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c27d82f0800.ost_server_uuid 50 [ 5807.999710] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c27d82f0800.ost_server_uuid in FULL state after 0 sec [ 5812.711175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.711973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5812.724312] Lustre: Skipped 3 previous similar messages [ 5816.707249] LustreError: 222781:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5816.710860] LustreError: 222781:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5816.901588] Lustre: server umount lustre-OST0000 complete [ 5819.873039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5819.877254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.882981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5825.770343] LustreError: 222983:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5825.777894] LustreError: 222983:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5825.886862] Lustre: server umount lustre-MDT0000 complete [ 5828.065700] LustreError: 219061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.084353] LustreError: 219061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5828.883070] LustreError: 219043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243214 with bad export cookie 10082990870736581873 [ 5828.889555] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5828.911118] LustreError: 219043:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5829.050609] LustreError: 223187:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5829.053919] LustreError: 223187:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5829.313109] Lustre: server umount lustre-MDT0001 complete [ 5839.410252] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 5841.660464] Key type lgssc unregistered [ 5841.933299] LNet: 223719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5841.950332] LNetError: 223719:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5841.975693] LNet: Removed LNI 192.168.206.127@tcp [ 5842.833087] Key type .llcrypt unregistered [ 5842.839354] Key type ._llcrypt unregistered [ 5865.916129] Key type ._llcrypt registered [ 5865.917836] Key type .llcrypt registered [ 5866.044194] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 5866.630650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5866.698314] alg: No test for adler32 (adler32-zlib) [ 5867.666525] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 5867.808054] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 5869.424205] Key type lgssc registered [ 5870.222725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5877.768762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5877.783523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.128039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5882.199875] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5887.154306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5887.363324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5889.684304] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5891.576327] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5896.423620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5896.709918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.782741] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5903.942461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49699 to 0x280000401:49729) [ 5911.183716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5911.308487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5911.516859] Lustre: lustre-OST0001: new disk, initializing [ 5911.530665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5911.741635] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5916.201624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5916.211752] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5916.335872] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5916.927161] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 5927.641964] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5932.567983] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5949.921135] LustreError: 228765:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5950.160643] Lustre: server umount lustre-MDT0000 complete [ 5951.972489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.999982] LustreError: 225408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.023695] LustreError: 225408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5954.339652] LustreError: 225388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243340 with bad export cookie 17906993834497067552 [ 5954.350177] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5954.365311] LustreError: 225388:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5954.501921] LustreError: 228966:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5954.512788] LustreError: 228966:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5954.706200] Lustre: server umount lustre-MDT0001 complete [ 5968.517457] LustreError: 229167:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5968.528511] LustreError: 229167:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5968.572361] Lustre: server umount lustre-OST0000 complete [ 5981.919085] LustreError: 229369:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5981.927217] LustreError: 229369:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5982.211226] Lustre: server umount lustre-OST0001 complete [ 5984.817189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5984.911118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5985.837596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 5985.889482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5986.921595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5987.063600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5988.147366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 5988.239572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5998.780633] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6005.534727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.865111] LustreError: 230779:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.958861] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6008.259967] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6011.362577] LustreError: 230779:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.264682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6014.507163] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6017.594544] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6020.278296] Lustre: 231891:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6025.843593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6030.257572] LustreError: 232246:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6030.277354] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49761) [ 6031.603461] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6035.431497] LustreError: 232507:0:(ldlm_lib.c:1179:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.281038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6039.514305] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6039.518523] Lustre: Skipped 1 previous similar message [ 6040.577663] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6043.901891] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6050.179121] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6052.609325] Lustre: 233741:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6065.636079] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6065.643261] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.654103] Lustre: Skipped 3 previous similar messages [ 6065.656928] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6066.158067] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6066.160874] Lustre: Skipped 2 previous similar messages [ 6071.281245] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6071.293390] Lustre: Skipped 4 previous similar messages [ 6071.462727] LustreError: 234046:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6071.473657] LustreError: 234046:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6071.531029] Lustre: server umount mylustre-MDT0000 complete [ 6074.337736] LustreError: 230753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243460 with bad export cookie 17906993834497070345 [ 6074.338757] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6074.351251] LustreError: 230753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6074.617935] Lustre: server umount mylustre-MDT0001 complete [ 6087.745749] LustreError: 234448:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6087.748865] LustreError: 234448:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6087.779052] Lustre: server umount mylustre-OST0000 complete [ 6100.821189] Lustre: server umount mylustre-OST0001 complete [ 6103.215120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6103.289512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6104.380540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6104.489398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6105.526818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6105.584382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6106.413582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6106.517609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6117.729658] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6125.449399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.793440] LustreError: 236060:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.892661] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6129.050814] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6137.112497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6141.032180] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6143.584807] Lustre: 237172:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6149.480150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6150.892214] LustreError: 237525:0:(ldlm_lib.c:1179:target_handle_connect()) tfs-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6150.906851] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49793) [ 6150.912572] LustreError: 237525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6154.859411] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6162.690939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6164.128762] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6167.835078] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6174.524886] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6177.673832] Lustre: 239018:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6188.518858] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6188.527827] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 6188.532502] Lustre: Skipped 4 previous similar messages [ 6188.538845] Lustre: Skipped 3 previous similar messages [ 6193.639787] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 6193.646339] Lustre: Skipped 3 previous similar messages [ 6194.020395] LustreError: 239268:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6194.023515] LustreError: 239268:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6194.069494] Lustre: server umount tfs-MDT0000 complete [ 6197.591106] LustreError: 236035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243583 with bad export cookie 17906993834497072235 [ 6197.595829] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6197.610559] LustreError: 236035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6198.091184] Lustre: server umount tfs-MDT0001 complete [ 6212.328679] Lustre: server umount tfs-OST0000 complete [ 6214.628859] Lustre: 224670:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761243584/real 1761243584] req@ffff90d3c1fb1500 x1846797405455872/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761243600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6214.647739] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.652817] Lustre: Skipped 2 previous similar messages [ 6216.136879] Lustre: server umount tfs-OST0001 complete [ 6218.971134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6219.062202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6220.101564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro [ 6220.204427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6221.131303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6221.222636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6222.123287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: ,errors=remount-ro [ 6222.199773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6236.019241] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6246.236814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.789966] LustreError: 241281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6246.814955] LustreError: 241281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6246.890706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6246.894249] Lustre: Skipped 3 previous similar messages [ 6250.449057] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6258.792787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6262.570961] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6265.522975] Lustre: 242393:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6272.516970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6279.135424] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6287.275909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6291.983617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49731 to 0x280000401:49825) [ 6292.016138] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6293.106971] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6299.972940] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6303.449724] Lustre: 244243:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6314.154800] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 14:21:38 (1761243698) [ 6330.849500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6330.852557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.869720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6334.882135] LustreError: 244611:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6334.885301] LustreError: 244611:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6335.025946] Lustre: server umount lustre-MDT0000 complete [ 6338.022199] LustreError: 241277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6338.039570] LustreError: 241277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6338.388681] LustreError: 242395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243724 with bad export cookie 17906993834497073901 [ 6338.390494] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6338.402469] LustreError: 242395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6343.138314] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.165893] Lustre: Skipped 4 previous similar messages [ 6344.870080] Lustre: server umount lustre-MDT0001 complete [ 6354.348748] Lustre: server umount lustre-OST0000 complete [ 6363.988460] Lustre: server umount lustre-OST0001 complete [ 6370.007864] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 6377.937567] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6386.665327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.243800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6397.724751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6403.949437] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6410.099473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6410.153886] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 6410.153886] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 6410.153886] [ 6410.173478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 6410.380489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6410.416808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6410.494813] Lustre: lustre-MDT0000: new disk, initializing [ 6410.610636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6410.614844] Lustre: Skipped 3 previous similar messages [ 6410.659980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6414.419889] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6424.410870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6424.481937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6424.551924] Lustre: 247403:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6424.617672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6424.621303] Lustre: Skipped 1 previous similar message [ 6424.671213] Lustre: lustre-MDT0001: new disk, initializing [ 6424.829408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6424.850097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6424.938455] Lustre: 248200:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6424.958597] Lustre: 248200:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6424.971395] Lustre: 248200:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6424.984961] Lustre: 248200:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6424.995653] Lustre: 248200:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6425.000994] Lustre: 248200:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6429.018937] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6438.720863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6438.782575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6439.053488] Lustre: lustre-OST0000: new disk, initializing [ 6439.063111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6441.035361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6441.056978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6441.225834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6444.701871] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6454.853904] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6454.909742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6455.019836] Lustre: lustre-OST0001: new disk, initializing [ 6455.029468] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6457.164441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6457.174986] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6457.234231] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6460.921424] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6472.548439] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 14:24:16 (1761243856) [ 6484.642565] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 14:24:28 (1761243868) [ 6502.883808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6502.890190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.898616] LustreError: Skipped 1 previous similar message [ 6502.899396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6502.904492] Lustre: Skipped 1 previous similar message [ 6502.912575] Lustre: Skipped 6 previous similar messages [ 6504.635200] LustreError: 250827:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6504.651293] LustreError: 250827:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6504.721763] Lustre: server umount lustre-OST0000 complete [ 6508.519532] LustreError: 249951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6508.537960] LustreError: 249951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6514.572361] Lustre: server umount lustre-MDT0000 complete [ 6519.149383] LustreError: 247395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761243905 with bad export cookie 17906993834497075455 [ 6519.151202] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6519.157629] LustreError: 247395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6525.751909] Lustre: server umount lustre-MDT0001 complete [ 6536.002626] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 6551.876713] Lustre: server umount lustre-OST0001 complete [ 6558.297431] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 6574.167315] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6583.685602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.531591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6594.940225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6601.012956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6607.518916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6607.576776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.768659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6607.804934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6607.868839] Lustre: lustre-MDT0000: new disk, initializing [ 6607.998331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6611.333783] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6620.122568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6620.189959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6620.257880] Lustre: 254315:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6620.490222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6620.622783] Lustre: 255115:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6620.648335] Lustre: 255115:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 6620.662992] Lustre: 255115:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6620.667608] Lustre: 255115:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6620.670285] Lustre: 255115:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6620.702425] Lustre: 255115:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6620.715672] Lustre: 255115:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6620.729551] Lustre: 255115:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6620.738467] Lustre: 255115:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6620.753577] Lustre: 255115:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6620.758206] Lustre: 255115:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6620.762104] Lustre: 255115:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6623.704749] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6630.856414] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6634.869293] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6642.697220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6642.777801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6642.959900] Lustre: lustre-OST0000: new disk, initializing [ 6642.962030] Lustre: Skipped 1 previous similar message [ 6642.964522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6642.967024] Lustre: Skipped 2 previous similar messages [ 6644.423977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6644.429647] Lustre: Skipped 1 previous similar message [ 6644.435852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6644.539863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6648.641761] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6655.798163] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6665.191883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6665.201645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6665.205662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6665.205671] Lustre: Skipped 11 previous similar messages [ 6665.221912] Lustre: Skipped 5 previous similar messages [ 6667.990640] LustreError: 256763:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6668.006369] LustreError: 256763:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6668.269279] Lustre: server umount lustre-OST0000 complete [ 6678.762459] Lustre: server umount lustre-MDT0000 complete [ 6682.145514] LustreError: 254310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244068 with bad export cookie 17906993834497076967 [ 6682.152334] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6682.167172] LustreError: 254310:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6682.623964] Lustre: server umount lustre-MDT0001 complete [ 6692.369419] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 6695.190651] Key type lgssc unregistered [ 6695.480687] LNet: 257698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6695.495225] LNetError: 257698:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6696.556151] LNet: Removed LNI 192.168.206.127@tcp [ 6697.436196] Key type .llcrypt unregistered [ 6697.438991] Key type ._llcrypt unregistered [ 6724.451256] Key type ._llcrypt registered [ 6724.452592] Key type .llcrypt registered [ 6724.684433] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6726.290563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6726.427327] alg: No test for adler32 (adler32-zlib) [ 6727.547263] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 6727.806799] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 6729.504302] Key type lgssc registered [ 6730.878315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6739.947278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6739.973863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.611713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.327791] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6751.209032] LustreError: 259142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6751.240142] LustreError: 259142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6754.014880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6754.646729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6759.006280] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6764.332245] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6767.512609] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6774.356586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6774.712504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6780.501909] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6785.981249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6786.126500] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6787.305218] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 6790.118218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6790.122069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6790.174942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6791.141322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6791.142127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6791.147599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.431257] LustreError: 261438:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6795.604093] Lustre: server umount lustre-OST0000 complete [ 6800.868656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6800.873085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.888626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6800.892458] Lustre: Skipped 1 previous similar message [ 6805.544289] LustreError: 261644:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6805.547429] LustreError: 261644:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6805.673997] Lustre: server umount lustre-MDT0000 complete [ 6806.498190] LustreError: 259142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6806.525930] LustreError: 259142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6808.974553] LustreError: 261340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244195 with bad export cookie 7529843959376142920 [ 6808.986100] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6808.995115] LustreError: 261340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6809.149291] LustreError: 261845:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6809.152263] LustreError: 261845:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6809.563737] Lustre: server umount lustre-MDT0001 complete [ 6818.689114] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 6820.721665] Key type lgssc unregistered [ 6820.923874] LNet: 262378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6820.932837] LNetError: 262378:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6820.981824] LNet: Removed LNI 192.168.206.127@tcp [ 6821.644987] Key type .llcrypt unregistered [ 6821.646637] Key type ._llcrypt unregistered [ 6836.373644] Key type ._llcrypt registered [ 6836.375172] Key type .llcrypt registered [ 6836.463644] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6837.279763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6837.315333] alg: No test for adler32 (adler32-zlib) [ 6838.368926] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 6838.639495] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 6840.328399] Key type lgssc registered [ 6841.348708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6850.488629] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6850.517201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6866.090233] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6871.200214] LustreError: 263552:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.127@tcp: failed processing log, type 4: rc = -110 [ 6897.952201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.760552] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 6907.327604] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6909.144884] LustreError: 264217:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 6909.329087] Lustre: server umount lustre-OST0000 complete [ 6920.908376] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 6922.915923] Key type lgssc unregistered [ 6923.138913] LNet: 265054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6923.145317] LNetError: 265054:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6923.158768] LNet: Removed LNI 192.168.206.127@tcp [ 6923.726096] Key type .llcrypt unregistered [ 6923.727588] Key type ._llcrypt unregistered [ 6953.909770] Key type ._llcrypt registered [ 6953.912223] Key type .llcrypt registered [ 6953.995980] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 6954.607713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6954.633541] alg: No test for adler32 (adler32-zlib) [ 6955.599774] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 6955.773969] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 6957.402147] Key type lgssc registered [ 6958.166677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6967.399046] loop0: detected capacity change from 0 to 204800000 [ 6967.500803] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6967.515028] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6967.686603] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6967.721951] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6968.288584] loop0: detected capacity change from 0 to 204800000 [ 6968.472874] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6968.482631] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6968.692493] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6968.724650] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6969.428933] loop0: detected capacity change from 0 to 204800000 [ 6969.527726] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6969.534840] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6969.606687] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6969.682122] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6970.414957] loop0: detected capacity change from 0 to 204800000 [ 6970.497595] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6970.655098] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6971.324489] loop0: detected capacity change from 0 to 204800000 [ 6971.343324] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6974.140072] loop0: detected capacity change from 0 to 204800000 [ 6974.181953] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6977.125648] loop0: detected capacity change from 0 to 204800000 [ 6977.155394] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6980.000816] loop0: detected capacity change from 0 to 204800000 [ 6980.031103] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 6983.165350] loop0: detected capacity change from 0 to 204800000 [ 6983.257611] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6983.331570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6983.357217] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6983.437000] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6983.446347] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 6983.474229] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6983.479840] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 6983.485647] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6983.490983] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 6983.496485] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 6983.503737] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6983.511990] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 6983.526646] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 6983.537437] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 6983.558821] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 6983.579946] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 6983.585972] Lustre: 268438:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 6989.387200] LustreError: 268706:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6989.470567] Lustre: server umount MGS complete [ 6990.406990] loop0: detected capacity change from 0 to 204800000 [ 6990.563956] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.931536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6990.981680] Lustre: lustre-MDT0000: Aborting client recovery [ 6991.899047] loop1: detected capacity change from 0 to 204800000 [ 6992.037105] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6992.217822] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6992.279799] Lustre: 268876:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6992.295498] Lustre: 268876:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6992.304465] Lustre: 268876:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 6992.332650] Lustre: 268876:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 6992.350531] Lustre: 268876:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6992.364078] Lustre: 268876:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 6992.380776] Lustre: 268876:0:(osd_scrub.c:1852:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6992.425560] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6992.778491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6992.872640] Lustre: lustre-MDT0001: Aborting client recovery [ 6992.908233] Lustre: 268906:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6992.914808] Lustre: 268906:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6992.919131] Lustre: 268906:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6992.923829] Lustre: 268906:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/89/0, punch: 0/0/0, quota 1/3/0 [ 6992.932755] Lustre: 268906:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6992.940677] Lustre: 268906:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6993.894374] loop2: detected capacity change from 0 to 204800000 [ 6994.087670] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6994.168909] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6994.214229] Lustre: 268962:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6994.226884] Lustre: 268962:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 6994.233629] Lustre: 268962:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 6994.245495] Lustre: 268962:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6994.252477] Lustre: 268962:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6994.259523] Lustre: 268962:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 6994.269056] Lustre: 268962:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6994.409440] Lustre: Found index 0 for lustre-OST0000, updating log [ 6994.572376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6994.610173] LustreError: 268818:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 3, retries 0, failed: rc = -108 [ 6994.617469] Lustre: lustre-OST0000: Aborting recovery [ 6995.606719] loop3: detected capacity change from 0 to 204800000 [ 6995.738699] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6995.858093] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6995.896630] Lustre: 269054:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 6995.907987] Lustre: 269054:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 6995.925503] Lustre: 269054:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 6995.952815] Lustre: 269054:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 6995.962805] Lustre: 269054:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 6995.997083] Lustre: 269054:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 6996.019781] Lustre: 269054:0:(osd_scrub.c:1852:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 6996.046960] Lustre: Found index 1 for lustre-OST0001, updating log [ 6996.197129] Lustre: lustre-OST0001: Aborting recovery [ 7000.305633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7000.309735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 7000.335339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 7000.351623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 7002.352717] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 7002.355158] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 7005.276619] LustreError: 268786:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 444us: rc = -2 [ 7005.777545] LustreError: 268782:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 794us: rc = -2 [ 7005.796255] LustreError: 268782:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 23 previous similar messages [ 7017.954482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.958570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7017.963552] Lustre: Skipped 1 previous similar message [ 7023.074842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7023.077851] Lustre: Skipped 3 previous similar messages [ 7023.845483] LustreError: 269405:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 7023.848270] LustreError: 269405:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7023.997235] Lustre: server umount lustre-MDT0000 complete [ 7024.717106] LustreError: 268767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244410 with bad export cookie 1676357735029371636 [ 7024.733441] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7028.194452] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.195647] LustreError: 268783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7028.196212] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7028.204888] Lustre: Skipped 3 previous similar messages [ 7028.218498] LustreError: 268783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7029.216588] LustreError: 268783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.238719] LustreError: 268783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7030.974347] LustreError: 269458:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 7030.979252] LustreError: 269458:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7031.310842] Lustre: server umount lustre-MDT0001 complete [ 7038.073023] LustreError: 269510:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 7038.077048] LustreError: 269510:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7038.126602] Lustre: server umount lustre-OST0000 complete [ 7053.280135] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7053.308820] LustreError: 269561:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7053.312563] LustreError: 269561:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7053.497070] Lustre: server umount lustre-OST0001 complete [ 7058.969912] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 14:34:03 (1761244443) [ 7080.742692] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 7088.556861] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7096.397697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7102.046306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7107.964824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7113.306352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7118.619356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7118.674797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7118.835256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7118.880565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7118.946325] Lustre: lustre-MDT0000: new disk, initializing [ 7119.098810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7119.102021] Lustre: Skipped 1 previous similar message [ 7119.134627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7122.377556] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7129.901047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7129.960166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7130.029980] Lustre: 273113:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7130.081251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7130.084223] Lustre: Skipped 1 previous similar message [ 7130.131741] Lustre: lustre-MDT0001: new disk, initializing [ 7130.201239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7130.238820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7130.244938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7130.319307] Lustre: 273909:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7130.323083] Lustre: 273909:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [ 7130.326514] Lustre: 273909:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7130.329861] Lustre: 273909:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7130.343635] Lustre: 273909:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7130.349024] Lustre: 273909:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7130.354106] Lustre: 273909:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7130.358659] Lustre: 273909:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7130.362912] Lustre: 273909:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7130.366716] Lustre: 273909:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7130.374764] Lustre: 273909:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7130.381589] Lustre: 273909:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 7132.531045] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7138.075755] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7141.463880] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7147.332977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7147.397277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7147.600938] Lustre: lustre-OST0000: new disk, initializing [ 7147.605181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7147.694581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7152.776655] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7154.237072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7154.246216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7154.353611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7159.277739] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7163.048690] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 7165.470820] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 7169.915801] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 7177.330895] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 7216.100879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7216.110783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.144115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7216.164807] Lustre: Skipped 4 previous similar messages [ 7220.664282] LustreError: 276438:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7220.666623] LustreError: 276438:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7220.778793] Lustre: server umount lustre-OST0000 complete [ 7226.339603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7226.341174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7226.345318] Lustre: Skipped 2 previous similar messages [ 7226.363699] Lustre: Skipped 1 previous similar message [ 7229.523241] Lustre: server umount lustre-MDT0000 complete [ 7231.458427] LustreError: 273120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7231.470108] LustreError: 273120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7232.003869] LustreError: 273105:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244618 with bad export cookie 1676357735029374408 [ 7232.006751] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7232.008563] LustreError: 273105:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7232.297256] Lustre: server umount lustre-MDT0001 complete [ 7237.925527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7240.691237] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7243.380679] LustreError: 277783:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7243.383397] LustreError: 277783:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7243.435291] Lustre: server umount MGS complete [ 7251.061197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7251.351846] LustreError: 278406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.378102] LustreError: 278418:0:(qmt_pool.c:1523:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7251.425322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7254.305180] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7256.552878] LustreError: 278406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7260.847924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7264.184150] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7269.047218] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7273.043556] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7280.355076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7286.836214] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7292.262616] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7322.592964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7322.597928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7322.606479] LustreError: Skipped 1 previous similar message [ 7322.620881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7322.635869] Lustre: Skipped 1 previous similar message [ 7326.135126] LustreError: 281065:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7326.141866] LustreError: 281065:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7326.278819] Lustre: server umount lustre-OST0000 complete [ 7332.833388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.843283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7332.847914] Lustre: Skipped 2 previous similar messages [ 7335.612961] Lustre: server umount lustre-MDT0000 complete [ 7337.956759] LustreError: 278406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7337.969285] LustreError: 278406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7340.213844] LustreError: 278386:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244726 with bad export cookie 1676357735029376277 [ 7340.216335] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7340.236687] LustreError: 278386:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7340.592384] Lustre: server umount lustre-MDT0001 complete [ 7348.048312] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 7350.079494] Key type lgssc unregistered [ 7350.323385] LNet: 282000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7350.328887] LNetError: 282000:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7350.347690] LNet: Removed LNI 192.168.206.127@tcp [ 7350.974600] Key type .llcrypt unregistered [ 7350.976318] Key type ._llcrypt unregistered [ 7380.621743] Key type ._llcrypt registered [ 7380.623848] Key type .llcrypt registered [ 7380.743460] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 7393.195654] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7394.036302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7394.077356] alg: No test for adler32 (adler32-zlib) [ 7395.256835] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 7395.445543] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 7397.104165] Key type lgssc registered [ 7398.208720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7405.503178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7411.712285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7417.715584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7424.802694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7430.860024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7430.934971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7430.949417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7432.148357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7432.180830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7432.238716] Lustre: lustre-MDT0000: new disk, initializing [ 7432.317113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7432.338191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7435.374329] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7443.607956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7443.671949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7443.726995] Lustre: 285923:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7443.763650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7443.766682] Lustre: Skipped 1 previous similar message [ 7443.816966] Lustre: lustre-MDT0001: new disk, initializing [ 7443.901887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7443.926824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7443.933468] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7444.010199] Lustre: 286721:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7444.019588] Lustre: 286721:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7444.025277] Lustre: 286721:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7444.030575] Lustre: 286721:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7444.036690] Lustre: 286721:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7444.040369] Lustre: 286721:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7446.957375] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7452.935538] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7456.233187] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7462.015215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7462.139771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7462.512077] Lustre: lustre-OST0000: new disk, initializing [ 7462.525652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7462.635927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7468.019099] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7469.133859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7469.142557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7469.296081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7475.227027] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7482.331135] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 7492.145126] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 7498.732576] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 7503.832272] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 7546.342396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7546.346397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7546.348621] LustreError: Skipped 1 previous similar message [ 7546.358168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7546.359731] Lustre: Skipped 1 previous similar message [ 7549.054435] LustreError: 289492:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7549.222316] Lustre: server umount lustre-OST0000 complete [ 7556.579660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7556.579726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.593871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.596566] Lustre: Skipped 1 previous similar message [ 7558.913836] LustreError: 289694:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7558.916969] LustreError: 289694:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7559.027854] Lustre: server umount lustre-MDT0000 complete [ 7562.172824] LustreError: 289331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761244948 with bad export cookie 13579338173314987039 [ 7562.183699] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7562.196561] LustreError: 289331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7562.303430] LustreError: 289899:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7562.308417] LustreError: 289899:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7562.605869] Lustre: server umount lustre-MDT0001 complete [ 7570.594437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.462901] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7578.554379] LustreError: 290840:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7578.567729] LustreError: 290840:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7578.678497] Lustre: server umount MGS complete [ 7588.621860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7588.937369] LustreError: 291463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7588.972266] LustreError: 291463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7588.978588] LustreError: 291475:0:(qmt_pool.c:1523:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 7589.036789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7591.651044] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7594.466146] LustreError: 291463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.473261] LustreError: 291463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7597.167704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7597.373573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7599.631058] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7604.517536] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7607.405685] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7612.005122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7612.232831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.942568] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7619.217396] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7645.153866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7645.157302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7645.170065] Lustre: Skipped 1 previous similar message [ 7645.186132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7646.599222] LustreError: 294125:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7646.629210] LustreError: 294125:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7646.797206] Lustre: server umount lustre-OST0000 complete [ 7653.345934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7653.347110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7653.352742] Lustre: Skipped 1 previous similar message [ 7653.359221] Lustre: Skipped 1 previous similar message [ 7655.729585] LustreError: 294326:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7655.731961] LustreError: 294326:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7655.823286] Lustre: server umount lustre-MDT0000 complete [ 7658.478708] LustreError: 291462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.501986] LustreError: 291462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7659.083421] LustreError: 291443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245045 with bad export cookie 13579338173314988915 [ 7659.085609] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.097461] LustreError: 291443:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7669.884114] Lustre: server umount lustre-MDT0001 complete [ 7677.307825] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 7679.713879] Key type lgssc unregistered [ 7679.954758] LNet: 295059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7679.966559] LNetError: 295059:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7679.979009] LNet: Removed LNI 192.168.206.127@tcp [ 7680.808693] Key type .llcrypt unregistered [ 7680.809974] Key type ._llcrypt unregistered [ 7697.127500] Key type ._llcrypt registered [ 7697.129413] Key type .llcrypt registered [ 7697.219948] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 7707.130768] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7707.771668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7707.830202] alg: No test for adler32 (adler32-zlib) [ 7708.824293] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 7708.976597] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 7710.656137] Key type lgssc registered [ 7711.614906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7718.204767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7723.061454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7729.377725] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 7734.419538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7739.390381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7739.492698] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7739.510368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.691067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7740.736465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7740.795474] Lustre: lustre-MDT0000: new disk, initializing [ 7740.880371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.905601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7743.745943] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 7751.176514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7751.238423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7751.453232] Lustre: lustre-OST0000: new disk, initializing [ 7751.463437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7751.466361] Lustre: Skipped 1 previous similar message [ 7751.535057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.930488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7754.935380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7754.974977] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7755.610831] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8448.640858] LustreError: 300048:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8448.996909] Lustre: server umount lustre-MDT0000 complete [ 8452.539045] LustreError: 298226:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245838 with bad export cookie 11503007192884627736 [ 8452.545721] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8462.854060] LustreError: 300249:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8462.863639] LustreError: 300249:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8463.130978] Lustre: server umount lustre-OST0000 complete [ 8482.483976] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 8490.427256] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8499.138463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8504.640805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8509.872789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8516.179914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8521.778262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8521.839416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.983249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8522.026896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8522.102566] Lustre: lustre-MDT0000: new disk, initializing [ 8522.185713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.220592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8525.465828] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8534.346662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8534.400917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8534.457859] Lustre: 303295:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8534.498793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8534.501446] Lustre: Skipped 1 previous similar message [ 8534.559602] Lustre: lustre-MDT0001: new disk, initializing [ 8534.653174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8534.698636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8534.714381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8534.816239] Lustre: 304090:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8534.826224] Lustre: 304090:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8534.851270] Lustre: 304090:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8534.862582] Lustre: 304090:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8534.877467] Lustre: 304090:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8534.886324] Lustre: 304090:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8537.920189] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8543.654437] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8546.574358] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8552.012579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8552.078046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8552.349110] Lustre: lustre-OST0000: new disk, initializing [ 8552.354281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8552.417561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8557.130215] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8558.160987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8558.166958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8558.246917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8563.030584] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8566.446806] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8566.650843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8569.709114] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8569.906223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8573.931927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8573.935376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8573.940928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8577.406646] LustreError: 306236:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8577.416961] LustreError: 306236:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8577.540513] Lustre: server umount lustre-OST0000 complete [ 8580.576812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8580.580097] LustreError: Skipped 1 previous similar message [ 8580.602481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.622101] Lustre: Skipped 1 previous similar message [ 8580.628337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8580.630981] Lustre: Skipped 1 previous similar message [ 8584.163249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8584.163743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8586.569787] LustreError: 306438:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8586.577160] LustreError: 306438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8586.672848] Lustre: server umount lustre-MDT0000 complete [ 8589.578383] LustreError: 303286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245975 with bad export cookie 11503007192887568597 [ 8589.582430] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8589.592889] LustreError: 303286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8589.825563] Lustre: server umount lustre-MDT0001 complete [ 8592.393583] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [ 8593.577949] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 14:59:38 (1761245978) [ 8598.830987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8599.117740] LustreError: 307211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8599.123569] LustreError: 307211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8599.184820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8602.416752] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8604.645688] LustreError: 307211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8609.150553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8609.440858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8612.472542] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8616.990446] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8620.004754] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8625.403193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8625.824068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.113075] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8637.774035] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8643.928973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8644.026117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8644.202377] Lustre: lustre-OST0001: new disk, initializing [ 8644.205667] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8644.331136] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8649.864914] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8650.824274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8650.835905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8650.870337] LustreError: 310132:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8650.878917] LustreError: 310132:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 8651.942402] LustreError: 310132:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [ 8651.961645] LustreError: 310132:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 8652.963186] LustreError: 310124:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8655.009782] LustreError: 310124:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8655.023732] LustreError: 310124:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages [ 8657.351831] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8657.551589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8659.118584] LustreError: 310124:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [ 8659.138904] LustreError: 310124:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [ 8660.311595] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9c27c5d87000.ost_server_uuid 50 [ 8661.699878] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9c27c5d87000.ost_server_uuid in FULL state after 0 sec [ 8664.841978] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c27c5d87000.ost_server_uuid 50 [ 8666.024471] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9c27c5d87000.ost_server_uuid in FULL state after 0 sec [ 8667.386895] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8681.447948] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8681.456648] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8681.464589] Lustre: Skipped 1 previous similar message [ 8681.469187] Lustre: Skipped 1 previous similar message [ 8686.569096] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8686.587932] Lustre: Skipped 3 previous similar messages [ 8686.673889] LustreError: 311034:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8686.684817] LustreError: 311034:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8686.767075] Lustre: server umount lustre-OST0001 complete [ 8691.681246] LustreError: 310133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8691.692411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.712824] LustreError: 310133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8691.722858] Lustre: Skipped 1 previous similar message [ 8696.804184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8696.806979] Lustre: Skipped 2 previous similar messages [ 8696.809886] LustreError: 308810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8697.173129] LustreError: 311235:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8697.183736] LustreError: 311235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8697.352315] Lustre: server umount lustre-OST0000 complete [ 8701.409469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8701.419595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.515752] Lustre: server umount lustre-MDT0000 complete [ 8707.042122] LustreError: 307211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8707.052807] LustreError: 307211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8709.382082] LustreError: 307190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246095 with bad export cookie 11503007192887569521 [ 8709.388317] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8709.398162] LustreError: 307190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8709.721667] Lustre: server umount lustre-MDT0001 complete [ 8717.693791] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 8720.319313] Key type lgssc unregistered [ 8720.589068] LNet: 312171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8720.593673] LNetError: 312171:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8721.638717] LNet: Removed LNI 192.168.206.127@tcp [ 8722.318901] Key type .llcrypt unregistered [ 8722.320546] Key type ._llcrypt unregistered [ 8741.830757] Key type ._llcrypt registered [ 8741.832599] Key type .llcrypt registered [ 8741.924857] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8742.955693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8742.995638] alg: No test for adler32 (adler32-zlib) [ 8744.012143] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 8744.184817] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 8745.816178] Key type lgssc registered [ 8746.663392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8752.620717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8752.647731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8754.011676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8756.958803] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8761.391196] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8764.388467] LustreError: 313550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8764.423720] LustreError: 313550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8768.020963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8768.324031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8771.626839] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8780.310108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8780.834753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8786.402755] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8791.116716] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8793.072850] LustreError: 315108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8796.832709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8797.046402] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8799.092920] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8799.102133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 8801.816330] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 8806.573976] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8811.492357] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8811.689324] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8911.852327] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8911.864477] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8911.888874] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8916.963449] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8916.967505] Lustre: Skipped 1 previous similar message [ 8917.835899] LustreError: 316984:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8917.896419] Lustre: server umount lustre-OST0001 complete [ 8922.080951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8922.086508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8922.088808] LustreError: 316147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8922.092782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8922.092791] Lustre: Skipped 1 previous similar message [ 8922.099987] Lustre: Skipped 2 previous similar messages [ 8922.108251] LustreError: 316147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8927.208373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8927.210383] LustreError: 315742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8927.225540] Lustre: Skipped 2 previous similar messages [ 8927.245469] LustreError: 315742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8928.123201] LustreError: 317185:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 8928.125977] LustreError: 317185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8928.293709] Lustre: server umount lustre-OST0000 complete [ 8932.323544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.326449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8932.327903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8932.348391] Lustre: Skipped 1 previous similar message [ 8937.335684] LustreError: 317387:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8937.347057] LustreError: 317387:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8937.446919] LustreError: 314726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.538720] Lustre: server umount lustre-MDT0000 complete [ 8940.548826] LustreError: 316882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246326 with bad export cookie 12947368396931685859 [ 8940.552919] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8940.564649] LustreError: 316882:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8940.691629] LustreError: 317588:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8940.694970] LustreError: 317588:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8940.873951] Lustre: server umount lustre-MDT0001 complete [ 8949.821435] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 8952.106337] Key type lgssc unregistered [ 8952.374452] LNet: 318120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8952.379558] LNetError: 318120:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8952.389692] LNet: Removed LNI 192.168.206.127@tcp [ 8953.066650] Key type .llcrypt unregistered [ 8953.068505] Key type ._llcrypt unregistered [ 8994.931307] Key type ._llcrypt registered [ 8994.934473] Key type .llcrypt registered [ 8995.039788] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 8996.323472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8996.381774] alg: No test for adler32 (adler32-zlib) [ 8997.538445] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 8997.771518] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 8999.456183] Key type lgssc registered [ 9000.381589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9008.375185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9008.391897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.767992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9013.423584] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9019.368522] LustreError: 320393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9019.388267] LustreError: 320393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9021.347037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9021.801760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9025.203640] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9027.615575] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9033.131586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9033.549314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9034.602583] LustreError: 321857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9034.616760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 9038.420930] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9039.846061] LustreError: 321855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9044.972801] LustreError: 321855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9045.389954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9045.569795] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9046.575667] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 9050.155854] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9057.901121] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9068.949866] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9087.464518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9087.468559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9087.478627] Lustre: Skipped 2 previous similar messages [ 9087.489471] Lustre: Skipped 3 previous similar messages [ 9090.027305] LustreError: 323809:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9090.135679] Lustre: server umount lustre-MDT0000 complete [ 9092.593513] LustreError: 320392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.614213] LustreError: 320392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9093.393558] LustreError: 321502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246479 with bad export cookie 3291890252209614054 [ 9093.402698] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9093.408180] LustreError: 321502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9093.518869] LustreError: 324010:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 9093.529046] LustreError: 324010:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9093.765953] Lustre: server umount lustre-MDT0001 complete [ 9107.981441] LustreError: 324211:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9107.994034] LustreError: 324211:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9108.034982] Lustre: server umount lustre-OST0000 complete [ 9121.420315] LustreError: 324413:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9121.423450] LustreError: 324413:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9121.610623] Lustre: server umount lustre-OST0001 complete [ 9136.495441] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9145.737945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9146.081982] LustreError: 325593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9146.160851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9149.602164] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9157.815824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9158.245852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9162.394622] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9165.603769] Lustre: 326703:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9172.982105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9178.995959] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9183.680221] LustreError: 327058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9183.701899] LustreError: 327058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9187.757845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9188.064120] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9188.077411] Lustre: Skipped 1 previous similar message [ 9191.100823] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 9191.106253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 9193.457928] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9200.554908] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9203.658081] Lustre: 328550:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9211.877840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9211.882200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9211.886022] Lustre: Skipped 3 previous similar messages [ 9211.892175] Lustre: Skipped 3 previous similar messages [ 9216.995854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9216.998712] Lustre: Skipped 3 previous similar messages [ 9217.473099] LustreError: 328653:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9217.494508] LustreError: 328653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9217.582432] Lustre: server umount lustre-MDT0000 complete [ 9220.498247] LustreError: 327873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246606 with bad export cookie 3291890252209615755 [ 9220.508474] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9220.510732] LustreError: 327873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9220.973187] Lustre: server umount lustre-MDT0001 complete [ 9234.954189] LustreError: 329057:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9234.961919] LustreError: 329057:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 9235.073222] Lustre: server umount lustre-OST0000 complete [ 9238.305089] Lustre: 319654:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761246608/real 1761246608] req@ffff90d4cb1fd880 x1846800687415296/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761246624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9238.329109] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.344256] Lustre: Skipped 1 previous similar message [ 9238.917604] Lustre: server umount lustre-OST0001 complete [ 9242.576026] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [ 9244.142391] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [ 9246.096354] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 15:10:30 (1761246630) [ 9266.142045] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9275.486479] loop0: detected capacity change from 0 to 1099511627776 [ 9276.113818] SGI XFS with ACLs, security attributes, quota, no debug enabled [ 9276.117080] *** VALIDATE xfs *** [ 9276.151406] XFS (loop0): Mounting V5 Filesystem [ 9276.173969] XFS (loop0): Ending clean mount [ 9277.389918] loop1: detected capacity change from 0 to 18691697672192 [ 9343.939795] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9358.705694] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [ 9365.867783] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9373.893724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9378.663955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9383.903137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9389.662092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9395.662918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9395.745659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9396.038802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9396.076721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9396.157635] Lustre: lustre-MDT0000: new disk, initializing [ 9396.269090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9396.291286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9399.909136] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9409.245214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9409.302977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9409.378212] Lustre: 333642:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9409.438859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9409.441694] Lustre: Skipped 1 previous similar message [ 9409.494457] Lustre: lustre-MDT0001: new disk, initializing [ 9409.647033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9409.656360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9409.752859] Lustre: 334438:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9409.764949] Lustre: 334438:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9409.775618] Lustre: 334438:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9409.786787] Lustre: 334438:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9409.802157] Lustre: 334438:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9409.807877] Lustre: 334438:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9413.826555] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9421.149877] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9424.172396] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9430.648201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9430.732403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9431.022795] Lustre: lustre-OST0000: new disk, initializing [ 9431.027925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9432.642761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9432.646949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9432.805355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9436.958630] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9444.562471] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9448.431395] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9448.665188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9452.451321] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9452.799384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9458.657217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9458.666854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9458.679897] LustreError: Skipped 1 previous similar message [ 9458.680924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9458.707437] Lustre: Skipped 1 previous similar message [ 9460.731491] LustreError: 336581:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9460.737713] LustreError: 336581:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 9461.129352] Lustre: server umount lustre-OST0000 complete [ 9465.825849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9465.834705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9465.861234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9465.863692] Lustre: Skipped 1 previous similar message [ 9471.259640] Lustre: server umount lustre-MDT0000 complete [ 9474.021959] LustreError: 333651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9474.048573] LustreError: 333651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9474.596357] LustreError: 333633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246860 with bad export cookie 3291890252209617309 [ 9474.601631] LustreError: 333633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9474.601981] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9475.276870] Lustre: server umount lustre-MDT0001 complete [ 9482.666875] XFS (loop0): Unmounting Filesystem [ 9487.171983] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 15:14:31 (1761246871) [ 9493.673580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9494.133344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9494.137654] Lustre: Skipped 2 previous similar messages [ 9497.716509] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9505.657619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9509.467199] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9514.324187] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9517.893500] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9523.622021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9529.144403] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9534.324418] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9549.796063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9549.807128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9549.812224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9549.825664] Lustre: Skipped 2 previous similar messages [ 9549.836707] Lustre: Skipped 3 previous similar messages [ 9560.035114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9560.090095] LustreError: 340145:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9560.098218] LustreError: 340145:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9560.325345] Lustre: server umount lustre-OST0000 complete [ 9569.884408] Lustre: server umount lustre-MDT0000 complete [ 9570.288110] LustreError: 338515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9570.293298] LustreError: 338515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9572.889849] LustreError: 337784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246958 with bad export cookie 3291890252209618233 [ 9572.893964] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9572.895931] LustreError: 337784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9573.228477] Lustre: server umount lustre-MDT0001 complete [ 9581.975418] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [ 9584.256341] Key type lgssc unregistered [ 9584.529852] LNet: 341082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9584.540111] LNetError: 341082:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9584.568080] LNet: Removed LNI 192.168.206.127@tcp [ 9585.266169] Key type .llcrypt unregistered [ 9585.270909] Key type ._llcrypt unregistered [ 9608.682506] Key type ._llcrypt registered [ 9608.684753] Key type .llcrypt registered [ 9608.780068] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [ 9609.755712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9609.774210] alg: No test for adler32 (adler32-zlib) [ 9610.848313] Lustre: Lustre: Build Version: 2.16.59_36_ga4fb14e [ 9611.049454] LNet: Added LNI 192.168.206.127@tcp [8/256/0/180] [ 9612.736241] Key type lgssc registered [ 9614.129736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9620.850096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9620.874146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9622.208271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9625.172464] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9632.353209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9632.664188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9636.515864] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9640.983873] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9643.777055] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9649.841820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9650.197659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9655.559281] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9659.942511] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9677.035496] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [ 9678.820362] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.835216] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9678.838284] Lustre: Skipped 1 previous similar message [ 9682.131605] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [ 9682.409554] LustreError: 344948:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9682.490495] Lustre: server umount lustre-MDT0001 complete [ 9683.943389] LustreError: 344736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9683.951063] LustreError: 344736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9684.607113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9687.251991] LustreError: 344789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.264859] LustreError: 344789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9689.060855] LustreError: 344736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9689.074818] LustreError: 344736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9689.551231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9689.633191] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9689.637734] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9689.674155] Lustre: 342617:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9689.910664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9692.879234] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [ 9695.210948] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9696.236682] LustreError: 345925:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [ 9696.240276] LustreError: 345925:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [ 9696.245162] LustreError: 342622:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 9696.278554] LustreError: 345928:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.206.127@tcp: cfg command failed: rc = -17 [ 9696.286590] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 9696.286590] [ 9999.585208] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.27@tcp (stopping) [10002.403032] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10002.409435] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10002.427360] Lustre: Skipped 1 previous similar message [10003.535917] LustreError: 354416:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10003.560170] LustreError: 354416:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10003.674759] Lustre: server umount lustre-MDT0001 complete [10004.687083] LustreError: 342626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10006.142584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10009.817357] LustreError: 342626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10009.860047] LustreError: 342626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10013.907673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10014.005713] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10014.013158] Lustre: Found index 1 for lustre-MDT0001, updating log [10014.087663] Lustre: 342618:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10014.381150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10018.618971] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10019.817629] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10023.424445] LustreError: 355397:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [10023.431930] LustreError: 355397:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [10023.445714] LustreError: 342622:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [10023.452666] LustreError: 342622:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [10023.496770] LustreError: 355400:0:(obd_config.c:689:class_setup()) Device 8 already setup (type osp) [10023.503800] LustreError: 355400:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.206.127@tcp: cfg command failed: rc = -17 [10023.530429] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.206.127@tcp [10023.530429] [10327.015938] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.019413] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10327.025885] Lustre: Skipped 1 previous similar message [10327.031391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10327.039975] Lustre: Skipped 3 previous similar messages [10337.249318] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10337.251695] Lustre: Skipped 4 previous similar messages [10338.272165] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10338.363325] LustreError: 363684:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10338.368594] LustreError: 363684:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10338.421360] Lustre: server umount lustre-MDT0001 complete [10340.831477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10342.375907] LustreError: 344789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.393549] LustreError: 344789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10348.204601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10348.298580] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10348.301497] Lustre: Found index 1 for lustre-MDT0001, updating log [10348.334515] Lustre: 342617:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10348.613054] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10352.892636] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10353.652817] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10355.701627] LustreError: 364634:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [10355.712602] LustreError: 364634:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [10355.721072] LustreError: 342622:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [10355.724405] LustreError: 342622:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [10355.838675] LustreError: 364646:0:(lod_lov.c:207:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [10355.853901] LustreError: 364646:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.206.127@tcp: cfg command failed: rc = -114 [10355.869317] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [10355.869317] [10676.193405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10676.197113] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10676.200816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10676.200828] Lustre: Skipped 1 previous similar message [10676.240844] Lustre: Skipped 2 previous similar messages [10677.865963] LustreError: 373071:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10677.869187] LustreError: 373071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10678.011691] Lustre: server umount lustre-MDT0000 complete [10681.207642] LustreError: 364252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248067 with bad export cookie 15847322898421399150 [10681.215084] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10681.314518] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.314616] LustreError: 342624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10681.342856] LustreError: 342624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10687.400903] LustreError: 373276:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10687.409153] LustreError: 373276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10687.652626] Lustre: server umount lustre-MDT0001 complete [10691.299299] Lustre: server umount lustre-OST0000 complete [10698.304670] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [10707.448697] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [10719.059620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10725.538028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10731.373250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10737.094889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10743.282266] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 15:35:27 (1761248127) [10749.872761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10749.957996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10750.189385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10750.248917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10750.311333] Lustre: lustre-MDT0000: new disk, initializing [10750.427944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10750.451935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10754.280464] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10763.616632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10763.692234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10763.811427] Lustre: 376012:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10763.872867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10763.878871] Lustre: Skipped 1 previous similar message [10763.998081] Lustre: lustre-MDT0001: new disk, initializing [10764.133588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10764.189197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10764.198267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10764.289241] Lustre: 376809:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [10764.299957] Lustre: 376809:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [10764.311748] Lustre: 376809:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [10764.321237] Lustre: 376809:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [10764.336428] Lustre: 376809:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [10764.353992] Lustre: 376809:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10768.150836] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10775.339897] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10778.412259] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10785.042365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10785.152306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10785.480035] Lustre: lustre-OST0000: new disk, initializing [10785.484339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10785.584484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10790.973786] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_default_debug -1 all [10794.131173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10794.150481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10794.239817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10798.840537] Lustre: DEBUG MARKER: oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10801.490988] Lustre: 376019:0:(osd_internal.h:1360:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 511, rollback = 2 [10801.495090] Lustre: 376019:0:(osd_internal.h:1360:osd_trans_exec_op()) Skipped 1 previous similar message [10801.498695] Lustre: 376019:0:(osd_handler.c:2081:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [10801.503127] Lustre: 376019:0:(osd_handler.c:2081:osd_trans_dump_creds()) Skipped 1 previous similar message [10801.510572] Lustre: 376019:0:(osd_handler.c:2088:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 5/511/0 [10801.517628] Lustre: 376019:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [10801.522640] Lustre: 376019:0:(osd_handler.c:2098:osd_trans_dump_creds()) write: 5/58/0, punch: 0/0/0, quota 1/3/0 [10801.526831] Lustre: 376019:0:(osd_handler.c:2098:osd_trans_dump_creds()) Skipped 1 previous similar message [10801.530670] Lustre: 376019:0:(osd_handler.c:2105:osd_trans_dump_creds()) insert: 5/102/3, delete: 0/0/0 [10801.535029] Lustre: 376019:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [10801.539193] Lustre: 376019:0:(osd_handler.c:2112:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10801.543094] Lustre: 376019:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [10807.780616] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10807.800550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10807.814146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10807.816774] Lustre: Skipped 4 previous similar messages [10811.554177] LustreError: 378457:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10811.560191] LustreError: 378457:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10811.691984] Lustre: server umount lustre-MDT0000 complete [10814.946231] LustreError: 376023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.965121] LustreError: 376023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10815.071054] LustreError: 376004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248201 with bad export cookie 15847322898421401117 [10815.081489] LustreError: MGC192.168.206.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10815.088451] LustreError: 376004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [10815.582401] Lustre: server umount lustre-MDT0001 complete [10829.566778] LustreError: 378862:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10829.569328] LustreError: 378862:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10829.809991] Lustre: server umount lustre-OST0000 complete [10857.046535] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing set_hostid [10865.610681] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing load_modules_local [10873.920076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10879.582627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10884.847760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10890.171539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10891.686680] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10600 sec ========= 15:37:55 (1761248275) [10893.189513] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:37:57 (1761248277) === [10895.964175] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:38:00 (1761248280) === [10932.895822] Lustre: DEBUG MARKER: oleg627-server.virtnet: executing unload_modules_local [10934.823240] Key type lgssc unregistered [10935.075871] LNet: 383973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10935.083198] LNetError: 383973:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10935.097450] LNet: Removed LNI 192.168.206.127@tcp [10935.822532] Key type .llcrypt unregistered [10935.826116] Key type ._llcrypt unregistered