[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496035221 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003065] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008382] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011118] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014079] *** VALIDATE selinux *** [ 0.022243] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026815] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027183] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029141] *** VALIDATE tmpfs *** [ 0.031292] *** VALIDATE proc *** [ 0.032243] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034293] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.041523] debug: unmapping init [mem 0xffffffff98859000-0xffffffff98860fff] [ 0.043868] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044742] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045028] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052319] rcu: Hierarchical SRCU implementation. [ 0.054492] smp: Bringing up secondary CPUs ... [ 0.055630] x86: Booting SMP configuration: [ 0.056030] .... node #0, CPUs: #1 #2 #3 [ 0.059441] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146281] node 0 deferred pages initialised in 82ms [ 0.149461] devtmpfs: initialized [ 0.150190] x86/mm: Memory block size: 128MB [ 0.152622] gcov: version magic: 0x41383552 [ 0.154347] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155059] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156266] pinctrl core: initialized pinctrl subsystem [ 0.157204] [ 0.157794] ************************************************************* [ 0.158014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159013] ** ** [ 0.160012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161012] ** ** [ 0.162014] ** This means that this kernel is built to expose internal ** [ 0.163014] ** IOMMU data structures, which may compromise security on ** [ 0.164011] ** your system. ** [ 0.165027] ** ** [ 0.166014] ** If you see this message and you are not debugging the ** [ 0.167015] ** kernel, report this immediately to your vendor! ** [ 0.168016] ** ** [ 0.169014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170016] ************************************************************* [ 0.171674] NET: Registered protocol family 16 [ 0.172431] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176010] cpuidle: using governor menu [ 0.177770] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181604] PCI: Using configuration type 1 for base access [ 0.183138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193129] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199453] cryptd: max_cpu_qlen set to 1000 [ 0.203298] ACPI: Added _OSI(Module Device) [ 0.205013] ACPI: Added _OSI(Processor Device) [ 0.206014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213177] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.219300] ACPI: Interpreter enabled [ 0.220057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221014] ACPI: Using IOAPIC for interrupt routing [ 0.222122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.225324] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234585] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.241081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246154] acpiphp: Slot [2] registered [ 0.247120] acpiphp: Slot [5] registered [ 0.249091] acpiphp: Slot [6] registered [ 0.250073] acpiphp: Slot [7] registered [ 0.251075] acpiphp: Slot [8] registered [ 0.252122] acpiphp: Slot [9] registered [ 0.253130] acpiphp: Slot [10] registered [ 0.254154] acpiphp: Slot [3] registered [ 0.256077] acpiphp: Slot [4] registered [ 0.257093] acpiphp: Slot [11] registered [ 0.258082] acpiphp: Slot [12] registered [ 0.259112] acpiphp: Slot [13] registered [ 0.260090] acpiphp: Slot [14] registered [ 0.261078] acpiphp: Slot [15] registered [ 0.262086] acpiphp: Slot [16] registered [ 0.263102] acpiphp: Slot [17] registered [ 0.265106] acpiphp: Slot [18] registered [ 0.266094] acpiphp: Slot [19] registered [ 0.268114] acpiphp: Slot [20] registered [ 0.269106] acpiphp: Slot [21] registered [ 0.271121] acpiphp: Slot [22] registered [ 0.272104] acpiphp: Slot [23] registered [ 0.273086] acpiphp: Slot [24] registered [ 0.274095] acpiphp: Slot [25] registered [ 0.276126] acpiphp: Slot [26] registered [ 0.277107] acpiphp: Slot [27] registered [ 0.279088] acpiphp: Slot [28] registered [ 0.280081] acpiphp: Slot [29] registered [ 0.281109] acpiphp: Slot [30] registered [ 0.282090] acpiphp: Slot [31] registered [ 0.284056] PCI host bridge to bus 0000:00 [ 0.285015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.290026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297235] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298819] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301940] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309508] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313926] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.316013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.318027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.321021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.323995] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.327914] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.330048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.333924] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.339014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.355077] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.359015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.362866] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.369025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.402016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.414104] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.421025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.446020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.455106] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.462017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.471020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.489018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.500978] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.510018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.519016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.540023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.551969] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.561029] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.581022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.619022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.633209] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.642019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.651019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.680027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.700110] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.703839] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.705354] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.708377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.711224] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.715198] iommu: Default domain type: Passthrough [ 0.716424] SCSI subsystem initialized [ 0.718104] ACPI: bus type USB registered [ 0.719112] usbcore: registered new interface driver usbfs [ 0.721067] usbcore: registered new interface driver hub [ 0.722107] usbcore: registered new device driver usb [ 0.724158] pps_core: LinuxPPS API ver. 1 registered [ 0.725009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.728058] PTP clock support registered [ 0.729155] EDAC MC: Ver: 3.0.0 [ 0.732067] PCI: Using ACPI for IRQ routing [ 0.733523] NetLabel: Initializing [ 0.734011] NetLabel: domain hash size = 128 [ 0.736014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.738077] NetLabel: unlabeled traffic allowed by default [ 0.740039] vgaarb: loaded [ 0.741251] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.749005] clocksource: Switched to clocksource kvm-clock [ 0.854310] VFS: Disk quotas dquot_6.6.0 [ 0.855399] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.857754] *** VALIDATE ramfs *** [ 0.858953] *** VALIDATE hugetlbfs *** [ 0.860340] pnp: PnP ACPI init [ 0.862561] pnp: PnP ACPI: found 6 devices [ 0.905615] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.908632] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.911587] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.914133] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915908] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918378] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.921511] NET: Registered protocol family 2 [ 0.924209] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.929101] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932824] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.937926] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.941417] TCP: Hash tables configured (established 65536 bind 65536) [ 0.944234] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.947081] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.949192] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951349] NET: Registered protocol family 1 [ 0.953646] RPC: Registered named UNIX socket transport module. [ 0.955261] RPC: Registered udp transport module. [ 0.956398] RPC: Registered tcp transport module. [ 0.958115] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.960245] NET: Registered protocol family 44 [ 0.961527] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.963312] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.964992] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.966698] PCI: CLS 0 bytes, default 64 [ 0.968172] Unpacking initramfs... [ 2.381645] debug: unmapping init [mem 0xffff95bf3cc54000-0xffff95bf3ffbffff] [ 2.385784] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.388181] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.391191] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.904141] Initialise system trusted keyrings [ 2.905995] Key type blacklist registered [ 2.907878] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.917419] zbud: loaded [ 2.920311] *** VALIDATE nfs *** [ 2.921322] *** VALIDATE nfs4 *** [ 2.923171] pstore: using deflate compression [ 2.926481] Platform Keyring initialized [ 3.035357] NET: Registered protocol family 38 [ 3.036893] Key type asymmetric registered [ 3.037882] Asymmetric key parser 'x509' registered [ 3.039386] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.042054] io scheduler mq-deadline registered [ 3.043365] io scheduler kyber registered [ 3.044685] io scheduler bfq registered [ 3.046608] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.049527] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.052219] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.055287] ACPI: Power Button [PWRF] [ 3.060900] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.067399] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.078917] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.085080] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.098677] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.126849] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.157232] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.162402] Non-volatile memory driver v1.3 [ 3.164123] Linux agpgart interface v0.103 [ 3.197813] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.200355] vda: detected capacity change from 0 to 74702848 [ 3.216041] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.218650] vdb: detected capacity change from 0 to 1073741824 [ 3.236521] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.238891] vdc: detected capacity change from 0 to 2621440000 [ 3.251703] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.254016] vdd: detected capacity change from 0 to 2621440000 [ 3.266209] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.268322] vde: detected capacity change from 0 to 4294967296 [ 3.280444] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.282521] vdf: detected capacity change from 0 to 4294967296 [ 3.287577] libphy: Fixed MDIO Bus: probed [ 3.295374] usbcore: registered new interface driver usbserial_generic [ 3.297341] usbserial: USB Serial support registered for generic [ 3.299295] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.302943] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.304564] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.306725] mousedev: PS/2 mouse device common for all mice [ 3.309543] rtc_cmos 00:05: RTC can wake from S4 [ 3.312293] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.315545] rtc_cmos 00:05: registered as rtc0 [ 3.316948] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.317642] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.321804] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.323183] intel_pstate: CPU model not supported [ 3.332124] hid: raw HID events driver (C) Jiri Kosina [ 3.334192] usbcore: registered new interface driver usbhid [ 3.336189] usbhid: USB HID core driver [ 3.337797] drop_monitor: Initializing network drop monitor service [ 3.340047] Initializing XFRM netlink socket [ 3.341821] NET: Registered protocol family 10 [ 3.344658] Segment Routing with IPv6 [ 3.345774] NET: Registered protocol family 17 [ 3.347412] mpls_gso: MPLS GSO support [ 3.351738] RAS: Correctable Errors collector initialized. [ 3.353303] AVX version of gcm_enc/dec engaged. [ 3.354457] AES CTR mode by8 optimization enabled [ 3.428459] sched_clock: Marking stable (3428426732, 0)->(4333075346, -904648614) [ 3.431304] registered taskstats version 1 [ 3.432901] Loading compiled-in X.509 certificates [ 3.434348] zswap: loaded using pool lzo/zbud [ 3.457098] Key type big_key registered [ 3.468724] Key type encrypted registered [ 3.470343] ima: No TPM chip found, activating TPM-bypass! [ 3.471899] ima: Allocated hash algorithm: sha1 [ 3.472777] ima: No architecture policies found [ 3.474689] evm: Initialising EVM extended attributes: [ 3.476413] evm: security.selinux [ 3.477593] evm: security.ima [ 3.478451] evm: security.capability [ 3.479236] evm: HMAC attrs: 0x1 [ 3.480899] rtc_cmos 00:05: setting system clock to 2026-09-05 00:52:16 UTC (1788569536) [ 3.487215] debug: unmapping init [mem 0xffffffff99803000-0xffffffff999fffff] [ 3.489927] debug: unmapping init [mem 0xffffffff98582000-0xffffffff98858fff] [ 3.499089] Write protecting the kernel read-only data: 28672k [ 3.501626] debug: unmapping init [mem 0xffffffff96c03000-0xffffffff96dfffff] [ 3.503198] debug: unmapping init [mem 0xffffffff97514000-0xffffffff975fffff] [ 3.527332] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.533931] systemd[1]: Detected virtualization kvm. [ 3.535566] systemd[1]: Detected architecture x86-64. [ 3.537137] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.561140] systemd[1]: No hostname configured. [ 3.562174] systemd[1]: Set hostname to . [ 3.563440] random: systemd: uninitialized urandom read (16 bytes read) [ 3.565234] systemd[1]: Initializing machine ID from random generator. [ 3.601443] random: ln: uninitialized urandom read (6 bytes read) [ 3.678319] random: systemd: uninitialized urandom read (16 bytes read) [ 3.681108] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.686315] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.689112] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.190904] device-mapper: uevent: version 1.0.3 [ 4.193221] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.258385] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 5.526439] random: fast init done [ 5.815599] scsi host0: ata_piix [ 5.893341] scsi host1: ata_piix [ 5.897314] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.901558] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.992975] random: crng init done [ 10.997530] random: 7 urandom warning(s) missed due to ratelimiting [ 12.806527] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.631073] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.381970] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.949308] SELinux: Disabled at runtime. [ 18.037216] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.050099] systemd[1]: Detected virtualization kvm. [ 18.054766] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.982652] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.987226] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.994298] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.998343] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.006081] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.018896] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.047589] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [ 20.451563] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.526388] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.855984] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.935741] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.392243] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.510030] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 28.185361] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 28.975178] NFS: Registering the id_resolver key type [ 28.977833] Key type id_resolver registered [ 28.982095] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg156-server login: [ 96.628008] hrtimer: interrupt took 8998324 ns [ 104.716097] libcfs: loading out-of-tree module taints kernel. [ 104.766366] Key type ._llcrypt registered [ 104.773560] Key type .llcrypt registered [ 104.866330] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 126.810280] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 128.868974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.899638] alg: No test for adler32 (adler32-zlib) [ 130.453721] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 131.476143] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 133.331155] Key type lgssc registered [ 135.537083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 157.736657] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 204.362435] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 218.425275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 218.474609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.816939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 219.863554] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 219.965535] Lustre: lustre-MDT0000: new disk, initializing [ 220.067513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.093532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 225.105429] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.517228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 240.644734] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 240.671057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 240.674573] Lustre: Skipped 1 previous similar message [ 240.759408] Lustre: lustre-MDT0001: new disk, initializing [ 240.836508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.867763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 240.892635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 245.649271] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.134718] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 261.462072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.876590] Lustre: lustre-OST0000: new disk, initializing [ 261.879786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 261.894755] Lustre: 8454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 262.037082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 263.755120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 263.760966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 263.868989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 269.187463] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 286.832722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 287.062900] Lustre: lustre-OST0001: new disk, initializing [ 287.069154] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 287.078857] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 287.189523] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 293.516335] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.017964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 296.038808] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 296.086423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 307.592619] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 315.330853] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 323.111056] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing check_logdir /tmp/testlogs/ [ 329.124455] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing yml_node [ 335.745570] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 338.622992] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 342.237547] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 343.833954] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 20:57:56 EDT 2026 [ 363.343698] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 364.874272] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 382.944974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.946814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 382.972621] Lustre: Skipped 1 previous similar message [ 382.983617] Lustre: Skipped 3 previous similar messages [ 387.316815] Lustre: server umount lustre-MDT0000 complete [ 392.178929] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569925 with bad export cookie 6997200463899586161 [ 392.188527] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 392.200658] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 393.186635] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 393.188378] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.207640] Lustre: Skipped 2 previous similar messages [ 393.213416] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 393.225121] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 397.219347] LustreError: 10306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 397.220222] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 397.250902] Lustre: Skipped 2 previous similar messages [ 398.622212] Lustre: server umount lustre-MDT0001 complete [ 402.704994] Lustre: server umount lustre-OST0000 complete [ 406.987650] Lustre: server umount lustre-OST0001 complete [ 413.994927] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 425.199404] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 464.762912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.055260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 465.090543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 465.183404] Lustre: lustre-MDT0000: new disk, initializing [ 465.286423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.317391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 470.181283] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 481.394361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.601205] Lustre: 15080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 481.708411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 481.711880] Lustre: Skipped 1 previous similar message [ 481.820859] Lustre: lustre-MDT0001: new disk, initializing [ 481.976985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 482.012894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 482.019831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 487.704743] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 498.463720] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 505.171842] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.583741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 512.791751] Lustre: lustre-OST0000: new disk, initializing [ 512.795734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 512.803368] Lustre: 17049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 512.922034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.705441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 514.723546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 514.864145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 520.574673] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 531.739983] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 537.123175] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 537.483067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 543.659265] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 543.943837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 550.882451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 550.903632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.915573] Lustre: Skipped 1 previous similar message [ 550.931208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 550.938028] Lustre: Skipped 1 previous similar message [ 552.090550] Lustre: server umount lustre-OST0000 complete [ 559.074372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 559.080167] LustreError: Skipped 1 previous similar message [ 559.088373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.112088] Lustre: Skipped 1 previous similar message [ 559.116747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.360165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 571.365988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.374553] Lustre: Skipped 4 previous similar messages [ 572.015951] Lustre: server umount lustre-MDT0000 complete [ 576.360561] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570109 with bad export cookie 6997200463899587694 [ 576.364256] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.374491] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 576.950869] Lustre: server umount lustre-MDT0001 complete [ 582.224622] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 21:01:54 (1788570114) [ 590.224536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.003442] LustreError: 19331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.026320] LustreError: 19331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 591.096185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 596.457506] LustreError: 19332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.517342] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 601.576985] LustreError: 19331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.651735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.279821] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 617.802652] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 623.882434] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 631.979324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 639.496408] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 648.546697] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 655.423207] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 662.498512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 662.500766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.535807] Lustre: Skipped 1 previous similar message [ 662.539472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 668.050685] Lustre: server umount lustre-MDT0000 complete [ 671.729847] LustreError: 19326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.745134] LustreError: 19326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 672.498248] LustreError: 19312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570205 with bad export cookie 6997200463899588576 [ 672.501310] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.508520] LustreError: 19312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 673.131324] Lustre: server umount lustre-MDT0001 complete [ 688.213725] Lustre: server umount lustre-OST0000 complete [ 713.766060] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 725.453747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.073632] LustreError: 23717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.205517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 726.216113] Lustre: Skipped 2 previous similar messages [ 731.180974] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 740.467207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.043700] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 747.972393] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 754.320993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 760.091675] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 769.121906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 769.371560] Lustre: lustre-OST0001: new disk, initializing [ 769.379451] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 769.387663] Lustre: 26031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 771.210747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 771.217106] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 771.335241] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 777.270952] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 788.096894] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 792.111181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 807.398629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 807.407051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 807.421915] Lustre: Skipped 2 previous similar messages [ 807.430918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 807.439138] Lustre: Skipped 3 previous similar messages [ 811.940619] LustreError: 27256:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff95be825fc100) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 811.982183] Lustre: server umount lustre-MDT0000 complete [ 813.546678] LustreError: 23712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.556739] LustreError: 23712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 815.033877] LustreError: 23699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570348 with bad export cookie 6997200463899589423 [ 815.042591] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 815.046601] LustreError: 23699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 815.376169] Lustre: server umount lustre-MDT0001 complete [ 829.882969] Lustre: server umount lustre-OST0000 complete [ 843.517946] Lustre: server umount lustre-OST0001 complete [ 862.381496] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 873.044626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.643103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.647967] Lustre: Skipped 3 previous similar messages [ 878.200270] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 879.077203] LustreError: 29066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.115486] LustreError: 29066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 887.514582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.568125] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 895.506207] Lustre: 30223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 902.426559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 909.515556] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 918.829775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 926.138835] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 934.652624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 938.642947] Lustre: 32103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 949.217859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 949.225357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 949.248166] Lustre: Skipped 3 previous similar messages [ 949.256574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 949.260702] Lustre: Skipped 4 previous similar messages [ 951.687687] Lustre: server umount lustre-MDT0000 complete [ 955.881061] LustreError: 31416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570488 with bad export cookie 6997200463899590711 [ 955.891447] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 956.208255] Lustre: server umount lustre-MDT0001 complete [ 970.862344] Lustre: server umount lustre-OST0000 complete [ 984.504922] Lustre: server umount lustre-OST0001 complete [ 993.700934] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 21:08:46 (1788570526) [ 1024.985408] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1035.498623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.863719] LustreError: 34908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.875200] LustreError: 34908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 1040.497410] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1049.492562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.234927] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1058.229484] Lustre: 36055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1065.079910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1072.386596] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1082.483093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.014884] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1098.297373] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1102.228957] Lustre: 37934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1108.453585] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.461651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.471539] Lustre: Skipped 6 previous similar messages [ 1108.482147] Lustre: Skipped 7 previous similar messages [ 1113.332774] Lustre: server umount lustre-MDT0000 complete [ 1117.472373] LustreError: 37936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570650 with bad export cookie 6997200463899591824 [ 1117.473804] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.495375] LustreError: 37936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1118.032554] Lustre: server umount lustre-MDT0001 complete [ 1132.368210] Lustre: server umount lustre-OST0000 complete [ 1134.580450] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788570651/real 1788570651] req@ffff95be823fd500 x1875451224361344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788570667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1136.641381] Lustre: server umount lustre-OST0001 complete [ 1146.194847] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 21:11:18 (1788570678) [ 1164.786667] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1177.357825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.182557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.195424] Lustre: Skipped 7 previous similar messages [ 1185.343861] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1196.504565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.460210] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1203.956728] Lustre: 41225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1211.690365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1220.115787] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1230.010567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1237.734646] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1246.379909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1250.248331] Lustre: 43102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1251.321365] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1251.328194] Lustre: Skipped 1 previous similar message [ 1255.910937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.935369] Lustre: Skipped 3 previous similar messages [ 1261.302744] Lustre: server umount lustre-MDT0000 complete [ 1265.578741] LustreError: 40057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570798 with bad export cookie 6997200463899592930 [ 1265.591277] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1265.599870] LustreError: 40057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1272.353983] Lustre: server umount lustre-MDT0001 complete [ 1283.041144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.724603] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x611b0cbfc8ef0d49 [ 1290.750989] Lustre: MGC192.168.201.156@tcp: Connection restored to 0@lo (at 0@lo) [ 1295.345752] LustreError: 43978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.370295] LustreError: 43978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 33 previous similar messages [ 1296.629620] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1300.478097] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1306.909155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.776026] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1312.785245] Lustre: Skipped 1 previous similar message [ 1313.091844] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1320.883673] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1322.801781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1329.173847] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1331.070599] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1337.827664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1343.683654] Lustre: server umount lustre-MDT0000 complete [ 1347.648062] LustreError: 40057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570880 with bad export cookie 6997200463899594057 [ 1347.659316] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1347.667269] LustreError: 40057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1348.282325] Lustre: server umount lustre-MDT0001 complete [ 1363.543128] Lustre: server umount lustre-OST0000 complete [ 1364.449633] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788570881/real 1788570881] req@ffff95bfc1b81180 x1875451224453760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788570897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1368.227434] Lustre: server umount lustre-OST0001 complete [ 1377.609214] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 21:15:09 (1788570909) [ 1398.036838] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1409.893736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.236558] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1425.446363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.846986] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1434.602791] Lustre: 48866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1442.522853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1449.477561] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1458.472837] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1465.698389] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1475.348592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1483.829943] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1500.138500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.152370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1500.153456] Lustre: Skipped 9 previous similar messages [ 1500.176162] Lustre: Skipped 23 previous similar messages [ 1506.024854] Lustre: server umount lustre-MDT0000 complete [ 1510.524824] LustreError: 50866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571043 with bad export cookie 6997200463899594855 [ 1510.525487] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1510.529533] LustreError: 50866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1510.761874] Lustre: server umount lustre-MDT0001 complete [ 1525.207784] Lustre: server umount lustre-OST0000 complete [ 1539.765478] Lustre: server umount lustre-OST0001 complete [ 1550.339771] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 21:18:02 (1788571082) [ 1552.976219] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1554.642728] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 21:18:07 (1788571087) [ 1567.670797] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 1577.663526] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1603.515935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.954816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1604.106483] Lustre: lustre-MDT0000: new disk, initializing [ 1604.236788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1609.430345] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1619.129685] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1625.697384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1626.113501] Lustre: lustre-OST0000: new disk, initializing [ 1626.120858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1626.131243] Lustre: Skipped 1 previous similar message [ 1626.142557] Lustre: 54950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1627.533849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1627.542497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1627.600351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1633.338832] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1643.660506] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1646.286830] Lustre: DEBUG MARKER: create test files [ 1683.940219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1697.248127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1697.670459] Lustre: server umount lustre-OST0000 complete [ 1702.801814] Lustre: server umount lustre-MDT0000 complete [ 1717.225488] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1720.418861] Key type lgssc unregistered [ 1720.853220] LNet: 56584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.863832] LNetError: 56584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.877989] LNet: Removed LNI 192.168.201.156@tcp [ 1722.179161] Key type .llcrypt unregistered [ 1722.186372] Key type ._llcrypt unregistered [ 1758.186360] Key type ._llcrypt registered [ 1758.189913] Key type .llcrypt registered [ 1758.362366] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1760.146300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.224712] alg: No test for adler32 (adler32-zlib) [ 1761.444364] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 1761.896458] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1763.792449] Key type lgssc registered [ 1765.075692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1774.511315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1774.528119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.272879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1782.455552] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1791.609923] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1799.595643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1800.027577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.087810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1801.197511] Lustre: 59321:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1801.203296] Lustre: 59321:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1801.213553] Lustre: 59321:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1801.220880] Lustre: 59321:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1801.232922] Lustre: 59321:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1801.244781] Lustre: 59321:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1806.993937] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1815.221714] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1820.484861] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1825.772486] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1835.051720] Lustre: server umount lustre-OST0000 complete [ 1849.489270] Lustre: server umount lustre-MDT0000 complete [ 1863.673717] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1867.096549] Key type lgssc unregistered [ 1867.346783] LNet: 60932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.357443] LNetError: 60932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.377858] LNet: Removed LNI 192.168.201.156@tcp [ 1868.664703] Key type .llcrypt unregistered [ 1868.668555] Key type ._llcrypt unregistered [ 1902.625759] Key type ._llcrypt registered [ 1902.627826] Key type .llcrypt registered [ 1902.766867] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1904.196823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1904.280015] alg: No test for adler32 (adler32-zlib) [ 1905.538694] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 1905.882931] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1907.624679] Key type lgssc registered [ 1909.148751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1918.711745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1918.745596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.309061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.849242] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1931.956457] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1940.737338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1941.255273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.395954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1945.516624] Lustre: 63773:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 514 < left 544, rollback = 9 [ 1945.524595] Lustre: 63773:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/1, destroy: 0/0/0 [ 1945.534301] Lustre: 63773:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1945.537609] Lustre: 63773:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1945.566386] Lustre: 63773:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1945.576704] Lustre: 63773:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1949.252810] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1958.301765] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1964.133774] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1976.298228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.302155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1976.313591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1979.468598] Lustre: server umount lustre-OST0000 complete [ 1983.876700] Lustre: server umount lustre-MDT0000 complete [ 1994.908544] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1998.365441] Key type lgssc unregistered [ 1998.667503] LNet: 65331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.676400] LNetError: 65331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.705539] LNet: Removed LNI 192.168.201.156@tcp [ 1999.466970] Key type .llcrypt unregistered [ 1999.467853] Key type ._llcrypt unregistered [ 2020.590783] Key type ._llcrypt registered [ 2020.605939] Key type .llcrypt registered [ 2020.892853] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 2037.538552] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2039.181276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.201246] alg: No test for adler32 (adler32-zlib) [ 2040.091615] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 2040.209556] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2041.824390] Key type lgssc registered [ 2042.937792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2080.410251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2080.471050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.010110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2082.058095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2082.253517] Lustre: lustre-MDT0000: new disk, initializing [ 2082.409966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.497563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2088.904786] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2100.871333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.091892] Lustre: 68486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2101.202627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2101.207069] Lustre: Skipped 1 previous similar message [ 2101.328883] Lustre: lustre-MDT0001: new disk, initializing [ 2101.450260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2101.512785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2101.529881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2106.169989] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2117.110593] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2122.964163] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2130.025794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2130.407664] Lustre: lustre-OST0000: new disk, initializing [ 2130.412709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2130.420256] Lustre: 70459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2130.552817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.799682] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2140.211290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2140.226393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2140.318404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2148.365917] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2153.523540] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2153.756872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2159.444547] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2159.729611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2165.732681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2165.748197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.776195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2166.245795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2166.261671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.721615] Lustre: server umount lustre-OST0000 complete [ 2173.409779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2173.420988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.437934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.444653] Lustre: Skipped 2 previous similar messages [ 2176.488899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.492694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2176.519587] Lustre: Skipped 1 previous similar message [ 2177.808779] Lustre: server umount lustre-MDT0000 complete [ 2182.063324] LustreError: 69285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571715 with bad export cookie 5823806708482008197 [ 2182.067415] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2182.074456] LustreError: 69285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2182.376257] Lustre: server umount lustre-MDT0001 complete [ 2190.789732] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 21:28:43 (1788571723) [ 2201.243384] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2240.063182] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 2249.228243] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2288.282269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.533930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2288.583374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2288.676971] Lustre: lustre-MDT0000: new disk, initializing [ 2288.807478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.835162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2294.833192] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2306.497974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.603770] Lustre: 76478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2306.658860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2306.665362] Lustre: Skipped 1 previous similar message [ 2306.746287] Lustre: lustre-MDT0001: new disk, initializing [ 2306.852263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2306.907373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2306.916784] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2311.999730] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2322.394776] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2328.965539] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2335.650205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.939339] Lustre: lustre-OST0000: new disk, initializing [ 2335.950295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2335.962594] Lustre: 78449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2336.069585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.866390] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2344.506251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2344.523233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2344.641040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2353.749395] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2359.248177] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2359.525238] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2365.288553] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2365.631793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2370.529942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2370.534446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.553522] LustreError: Skipped 1 previous similar message [ 2370.557297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2370.591211] Lustre: Skipped 1 previous similar message [ 2375.651454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2375.661975] Lustre: Skipped 2 previous similar messages [ 2382.304162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2382.673412] Lustre: server umount lustre-OST0000 complete [ 2388.961555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2388.978439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.996602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.999102] Lustre: Skipped 2 previous similar messages [ 2401.248403] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2401.634975] Lustre: server umount lustre-MDT0000 complete [ 2405.662021] LustreError: 76469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571938 with bad export cookie 5823806708482009079 [ 2405.666720] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2405.673801] LustreError: 76469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2406.344570] Lustre: server umount lustre-MDT0001 complete [ 2416.096155] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 21:32:28 (1788571948) [ 2423.594160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2424.250563] LustreError: 80864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.269158] LustreError: 80864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2424.353755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.415742] LustreError: 80865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.579333] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2434.531953] LustreError: 80864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.680970] LustreError: 80865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2440.724057] LustreError: 80865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2442.608325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.519036] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2458.340114] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2464.790026] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2472.720312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2473.122251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.125349] Lustre: Skipped 1 previous similar message [ 2480.104488] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2489.386646] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2493.269709] Lustre: 83540:0:(genops.c:1795:obd_export_evict_by_uuid()) MGS: evicting 533d750e-7047-4b20-b34d-17c534dec9e6 at adminstrative request [ 2493.930026] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.939561] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2503.141097] Lustre: MGS: Client 533d750e-7047-4b20-b34d-17c534dec9e6 (at 0@lo) reconnecting [ 2503.151386] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x50d25158739b6b69 to 0x50d25158739b6e02 [ 2503.172863] Lustre: MGC192.168.201.156@tcp: Connection restored to 0@lo (at 0@lo) [ 2531.436940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2531.734890] Lustre: lustre-OST0001: new disk, initializing [ 2531.740765] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2531.751479] Lustre: 83933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2537.605641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2537.619961] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2537.723624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2538.856357] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2551.298290] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2558.436994] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2558.450562] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.469811] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2558.478333] Lustre: Skipped 2 previous similar messages [ 2558.501606] Lustre: Skipped 7 previous similar messages [ 2561.555550] Lustre: server umount lustre-OST0001 complete [ 2563.584946] LustreError: 82691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.610884] LustreError: 82691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2572.790827] Lustre: server umount lustre-OST0000 complete [ 2584.725215] Lustre: server umount lustre-MDT0000 complete [ 2588.144084] LustreError: 80864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.161062] LustreError: 80864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2589.321769] LustreError: 80845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572122 with bad export cookie 5823806708482010626 [ 2589.326623] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.331205] LustreError: 80845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2589.601533] Lustre: server umount lustre-MDT0001 complete [ 2600.962663] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 21:35:32 (1788572132) [ 2602.877545] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2605.020984] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 21:35:36 (1788572136) [ 2607.178791] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2609.272686] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 21:35:41 (1788572141) [ 2611.151470] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2613.439333] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 21:35:45 (1788572145) [ 2615.398686] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2617.439923] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 21:35:49 (1788572149) [ 2619.173935] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2626.186816] loop: module loaded [ 2626.202209] loop0: detected capacity change from 0 to 4294967296 [ 2626.259592] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2628.354608] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2632.316873] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2637.420792] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2664.430865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2664.452413] LustreError: 88059:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2664.456700] LustreError: 88059:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2664.461451] LustreError: 88059:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2664.467832] LustreError: 88059:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2685.494461] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 2698.245315] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2741.351879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.565180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2741.584418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2741.699954] Lustre: lustre-MDT0000: new disk, initializing [ 2741.786282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2741.797565] Lustre: Skipped 1 previous similar message [ 2741.824717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2747.675470] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2759.962487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.241400] Lustre: 90928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2760.607455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2765.807930] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2775.458933] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2782.351207] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2790.225204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2790.638663] Lustre: 92898:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2797.521638] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2797.669539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2797.847093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2806.814692] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2811.907414] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2812.139578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2817.095499] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2817.344982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2823.649880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2823.656464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.663038] LustreError: Skipped 3 previous similar messages [ 2823.666074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2823.666082] Lustre: Skipped 7 previous similar messages [ 2823.685501] Lustre: Skipped 5 previous similar messages [ 2825.887584] Lustre: server umount lustre-OST0000 complete [ 2836.845640] Lustre: server umount lustre-MDT0000 complete [ 2839.010991] LustreError: 90939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2839.030949] LustreError: 90939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2841.435513] LustreError: 90920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572374 with bad export cookie 5823806708482011242 [ 2841.442169] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.455267] LustreError: 90920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2841.822250] Lustre: server umount lustre-MDT0001 complete [ 2852.367210] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 21:39:44 (1788572384) [ 2860.725726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.867486] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2871.779691] LustreError: 95309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2871.795610] LustreError: 95309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2876.704621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.058859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2877.066390] Lustre: Skipped 3 previous similar messages [ 2882.464515] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2891.774289] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2897.826532] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2906.518441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2914.412108] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2923.873482] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2929.143365] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2932.800661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2938.681339] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2938.977372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2946.394184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2946.642629] Lustre: lustre-OST0001: new disk, initializing [ 2946.645734] Lustre: Skipped 2 previous similar messages [ 2946.650213] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2946.655687] Lustre: Skipped 3 previous similar messages [ 2946.662347] Lustre: 98704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2952.354349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2952.363854] Lustre: Skipped 2 previous similar messages [ 2952.376907] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2952.493989] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2954.569072] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2965.656902] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2971.656732] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2971.963745] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2978.085842] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2978.378216] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2998.356590] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3025.162953] Lustre: Failing over lustre-MDT0000 [ 3025.381743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3025.389644] LustreError: Skipped 1 previous similar message [ 3025.393809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.407900] Lustre: Skipped 2 previous similar messages [ 3025.411658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.415780] Lustre: Skipped 5 previous similar messages [ 3025.714793] Lustre: server umount lustre-MDT0000 complete [ 3026.157747] LustreError: 95305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3026.200817] LustreError: 95305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3045.926996] Lustre: 66709:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572562/real 1788572562] req@000000004922bec1 x1875453226017920/t0(0) o400->MGC192.168.201.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788572578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3045.973223] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3051.294700] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3051.299677] LDISKFS-fs (dm-0): recovery complete [ 3051.308505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.636492] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x50d25158739c3884 [ 3056.663123] Lustre: MGC192.168.201.156@tcp: Connection restored to 0@lo (at 0@lo) [ 3057.158488] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3059.917972] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3059.923531] Lustre: 95304:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3061.984782] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3062.249140] Lustre: 97607:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 3062.265801] Lustre: 97607:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 3062.279488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3062.415613] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3062.728068] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3063.081668] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3063.101820] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 3063.433097] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3063.436690] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 1 previous similar message [ 3063.457107] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3063.481104] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 7 previous similar messages [ 3064.123254] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3064.143224] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 3064.464056] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3064.476663] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 2 previous similar messages [ 3065.455187] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3065.466814] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 5 previous similar messages [ 3066.425185] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3066.442631] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 3066.768764] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3066.789927] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 6 previous similar messages [ 3069.666423] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3069.675396] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 12 previous similar messages [ 3070.642249] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3070.652222] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 3070.968154] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3070.975238] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 12 previous similar messages [ 3077.680778] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3077.687436] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 24 previous similar messages [ 3078.652350] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3078.665272] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 3078.984119] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3078.988929] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 24 previous similar messages [ 3093.890640] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3093.907535] Lustre: 101555:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 50 previous similar messages [ 3094.858299] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3094.868391] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 50 previous similar messages [ 3095.194344] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3095.204798] LustreError: 101555:0:(ldlm_lib.c:2698:replay_request_or_update()) Skipped 50 previous similar messages [ 3119.072193] Lustre: 101555:0:(ldlm_lib.c:2394:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3119.083548] Lustre: 101555:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3119.089866] Lustre: 101555:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 3e893c7e-43ed-4def-8146-374c68fb539a@192.168.201.56@tcp [ 3119.120570] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3119.151332] Lustre: 101555:0:(ldlm_lib.c:1914:abort_req_replay_queue()) @@@ aborted: req@00000000c2d262d6 x1875453210071808/t0(8589934774) o35->3e893c7e-43ed-4def-8146-374c68fb539a@192.168.201.56@tcp:312/0 lens 392/0 e 0 to 0 dl 1788572662 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3119.194293] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3119.194538] Lustre: lustre-MDT0000: Denying connection for new client 3e893c7e-43ed-4def-8146-374c68fb539a (at 192.168.201.56@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3119.238463] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3119.290748] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3119.292731] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3119.330775] Lustre: Skipped 2 previous similar messages [ 3119.368732] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:46 to 0x2c0000401:545) [ 3119.371219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:79 to 0x280000401:577) [ 3144.643544] Lustre: server umount lustre-OST0000 complete [ 3154.404227] LustreError: 97132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.421937] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3154.442256] LustreError: 97132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 3154.461024] Lustre: Skipped 9 previous similar messages [ 3154.871445] Lustre: server umount lustre-OST0001 complete [ 3159.522953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.547135] Lustre: Skipped 8 previous similar messages [ 3173.344120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3173.577825] Lustre: server umount lustre-MDT0000 complete [ 3177.973358] LustreError: 95289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572710 with bad export cookie 5823806708482062468 [ 3177.978452] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3177.982982] LustreError: 95289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3178.438940] Lustre: server umount lustre-MDT0001 complete [ 3188.867751] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 21:45:20 (1788572720) [ 3197.905464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.921124] Lustre: *** cfs_fail_loc=197, val=0*** [ 3213.602223] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3218.724849] LustreError: 103533:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 3223.934630] Lustre: *** cfs_fail_loc=197, val=0*** [ 3223.939594] Lustre: Skipped 65 previous similar messages [ 3244.512524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.529971] Lustre: Skipped 3 previous similar messages [ 3250.206738] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3258.747207] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3261.110629] Lustre: server umount lustre-OST0000 complete [ 3272.251457] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 21:46:44 (1788572804) [ 3295.616575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.930418] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3312.162663] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3317.605858] LustreError: 105186:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3317.620129] LustreError: 105186:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3317.630104] LustreError: 105186:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 3318.119207] Lustre: server umount lustre-MDT0000 complete [ 3357.764639] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 3370.680047] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3410.220474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.546045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3410.589051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3410.706108] Lustre: lustre-MDT0000: new disk, initializing [ 3410.800592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3415.868400] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3427.382830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3427.672990] Lustre: 109364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3427.949586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3433.433248] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3443.802829] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3450.395299] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3457.400224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3457.822957] Lustre: 111328:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3465.051963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3465.845740] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3477.181320] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3483.552720] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3483.839691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3489.345949] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3489.671318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3495.912489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3495.914539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.916026] LustreError: Skipped 1 previous similar message [ 3495.917087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3495.917093] Lustre: Skipped 6 previous similar messages [ 3495.955716] Lustre: Skipped 1 previous similar message [ 3506.144167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3506.308263] Lustre: server umount lustre-OST0000 complete [ 3525.088139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3525.379249] Lustre: server umount lustre-MDT0000 complete [ 3526.629057] LustreError: 109374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.648453] LustreError: 109374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3529.758374] LustreError: 111339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573062 with bad export cookie 5823806708482073101 [ 3529.769349] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3529.771351] LustreError: 111339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3530.159670] Lustre: server umount lustre-MDT0001 complete [ 3540.740326] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 21:51:12 (1788573072) [ 3551.152466] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3554.066959] Key type lgssc unregistered [ 3554.440498] LNet: 113824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3554.456448] LNetError: 113824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3554.470973] LNet: Removed LNI 192.168.201.156@tcp [ 3555.355653] Key type .llcrypt unregistered [ 3555.357277] Key type ._llcrypt unregistered [ 3575.802925] Key type ._llcrypt registered [ 3575.805588] Key type .llcrypt registered [ 3575.918952] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 3592.544445] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3593.771427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3594.004569] alg: No test for adler32 (adler32-zlib) [ 3594.969159] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 3595.257797] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3596.979988] Key type lgssc registered [ 3598.427451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3640.530658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3640.564111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.937032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3641.997376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3642.070289] Lustre: lustre-MDT0000: new disk, initializing [ 3642.156317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.169914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3647.083334] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3657.143785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.386834] Lustre: lustre-OST0000: new disk, initializing [ 3657.391785] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3657.401028] Lustre: Skipped 1 previous similar message [ 3657.407518] Lustre: 118004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3657.480717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.761059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3659.770836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3659.802588] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3663.706186] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3675.020540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3675.251409] Lustre: lustre-OST0001: new disk, initializing [ 3675.263766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3675.283866] Lustre: 118867:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3675.379638] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3680.938640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3680.956358] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3681.004109] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3681.818825] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3691.489545] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3691.493931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3691.505740] Lustre: Skipped 1 previous similar message [ 3691.540743] Lustre: Skipped 1 previous similar message [ 3696.613474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.618948] Lustre: Skipped 1 previous similar message [ 3697.547214] Lustre: server umount lustre-MDT0000 complete [ 3705.284360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3711.025844] LustreError: 117190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573244 with bad export cookie 4162806837592773119 [ 3711.046152] LustreError: 117190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3711.050357] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3711.112396] Lustre: server umount lustre-OST0000 complete [ 3715.754529] Lustre: server umount lustre-OST0001 complete [ 3725.340856] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 21:54:17 (1788573257) [ 3748.098867] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 21:54:40 (1788573280) [ 3758.638786] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3788.649543] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 21:55:19 (1788573319) [ 3836.638946] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 3850.353538] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3891.165181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.406451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3891.433248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3891.492543] Lustre: lustre-MDT0000: new disk, initializing [ 3891.544183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.556524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3896.556644] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3908.110872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3908.227627] Lustre: 126643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3908.285745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3908.288700] Lustre: Skipped 1 previous similar message [ 3908.348476] Lustre: lustre-MDT0001: new disk, initializing [ 3908.447361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3908.484037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3908.504508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3912.454415] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3923.318491] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3929.232611] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3938.118912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3938.411298] Lustre: lustre-OST0000: new disk, initializing [ 3938.413747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3938.417706] Lustre: 128610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3938.476744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3944.504619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3944.523966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3944.631142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3945.762603] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3956.773398] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3962.661694] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3962.817108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3967.373335] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3967.548393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3970.534022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3970.538555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.540709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3970.550362] LustreError: Skipped 1 previous similar message [ 3970.579681] Lustre: Skipped 1 previous similar message [ 3975.651627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3975.661576] Lustre: Skipped 1 previous similar message [ 3980.773948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.783659] Lustre: Skipped 1 previous similar message [ 3984.352130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3984.495298] Lustre: server umount lustre-OST0000 complete [ 3990.497265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3990.500491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.548971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.296496] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4003.649970] Lustre: server umount lustre-MDT0000 complete [ 4008.818363] LustreError: 126633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573541 with bad export cookie 4162806837592774337 [ 4008.836311] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.517048] Lustre: server umount lustre-MDT0001 complete [ 4019.415192] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 21:59:11 (1788573551) [ 4026.156629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4026.461467] LustreError: 131025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4026.481942] LustreError: 131025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4026.528148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4031.567944] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4031.970367] LustreError: 131026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.068308] LustreError: 131025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.292512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4045.483646] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4052.454945] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4058.214310] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4065.748455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4066.017234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4066.021801] Lustre: Skipped 1 previous similar message [ 4073.226799] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4082.394510] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4090.314484] Lustre: *** cfs_fail_loc=159, val=0*** [ 4107.014626] Lustre: lustre-MDT0000: Client 03bb3d1b-e256-4c31-aa37-32f5f48d1d58 (at 192.168.201.56@tcp) reconnecting [ 4109.065884] Lustre: *** cfs_fail_loc=159, val=0*** [ 4109.077155] Lustre: Skipped 6 previous similar messages [ 4124.423242] Lustre: lustre-MDT0000: Client 03bb3d1b-e256-4c31-aa37-32f5f48d1d58 (at 192.168.201.56@tcp) reconnecting [ 4131.755120] Lustre: server umount lustre-OST0000 complete [ 4136.932526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.936732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4136.953393] Lustre: Skipped 2 previous similar messages [ 4136.969992] Lustre: Skipped 7 previous similar messages [ 4145.371495] Lustre: server umount lustre-MDT0000 complete [ 4147.168324] Lustre: 115207:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788573664/real 1788573664] req@00000000e3c13c74 x1875454856377216/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788573680 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4147.171970] LustreError: 131020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.194580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.222680] LustreError: 131020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4148.919300] LustreError: 131007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573681 with bad export cookie 4162806837592775226 [ 4148.922191] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4148.932686] LustreError: 131007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4149.395596] Lustre: server umount lustre-MDT0001 complete [ 4160.008915] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4162.954630] Key type lgssc unregistered [ 4163.206693] LNet: 134821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4163.210927] LNetError: 134821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4163.230976] LNet: Removed LNI 192.168.201.156@tcp [ 4164.062674] Key type .llcrypt unregistered [ 4164.063883] Key type ._llcrypt unregistered [ 4186.916959] Key type ._llcrypt registered [ 4186.920760] Key type .llcrypt registered [ 4187.060695] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4188.426670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4188.498072] alg: No test for adler32 (adler32-zlib) [ 4189.888713] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 4190.253088] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4192.008153] Key type lgssc registered [ 4193.304553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4201.765810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4201.801894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.306319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.840521] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4213.736266] LustreError: 136233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.753678] LustreError: 136233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4216.499048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.933777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4221.834945] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4229.973441] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4236.395494] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4243.285336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4243.706920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4248.835307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4251.738977] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4260.549935] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4269.838861] Lustre: *** cfs_fail_loc=159, val=0*** [ 4286.203520] Lustre: lustre-MDT0000: Client 2c466a44-9fc7-43da-bcd2-1c3d5ea52252 (at 192.168.201.56@tcp) reconnecting [ 4288.521519] Lustre: *** cfs_fail_loc=159, val=0*** [ 4305.169589] Lustre: lustre-MDT0001: Client 2c466a44-9fc7-43da-bcd2-1c3d5ea52252 (at 192.168.201.56@tcp) reconnecting [ 4306.995598] Lustre: *** cfs_fail_loc=159, val=0*** [ 4306.998854] Lustre: Skipped 3 previous similar messages [ 4322.556450] Lustre: lustre-MDT0001: Client 2c466a44-9fc7-43da-bcd2-1c3d5ea52252 (at 192.168.201.56@tcp) reconnecting [ 4340.029962] Lustre: 139295:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4344.482919] Lustre: *** cfs_fail_loc=159, val=0*** [ 4344.505099] Lustre: Skipped 4 previous similar messages [ 4360.476240] Lustre: lustre-MDT0000: Client 966323c5-b97e-4ee6-b6c7-352185879aad (at 192.168.201.56@tcp) reconnecting [ 4362.680583] Lustre: *** cfs_fail_loc=159, val=0*** [ 4362.688443] Lustre: Skipped 14 previous similar messages [ 4378.446657] Lustre: lustre-MDT0000: Client 966323c5-b97e-4ee6-b6c7-352185879aad (at 192.168.201.56@tcp) reconnecting [ 4381.130919] Lustre: 139587:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4389.347021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.348983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4389.363460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4389.409759] Lustre: Skipped 1 previous similar message [ 4389.440558] Lustre: Skipped 1 previous similar message [ 4390.690749] Lustre: server umount lustre-OST0000 complete [ 4396.009600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4396.022260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.035581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.586803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.600616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.614309] Lustre: Skipped 1 previous similar message [ 4402.035841] Lustre: server umount lustre-MDT0000 complete [ 4404.708040] LustreError: 136228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.733353] LustreError: 136228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4405.861098] LustreError: 139053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573938 with bad export cookie 16363274836796365448 [ 4405.867959] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4405.870292] LustreError: 139053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4406.327573] Lustre: server umount lustre-MDT0001 complete [ 4418.459720] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4421.325942] Key type lgssc unregistered [ 4421.794705] LNet: 140645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4421.820886] LNetError: 140645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4421.838847] LNet: Removed LNI 192.168.201.156@tcp [ 4422.818642] Key type .llcrypt unregistered [ 4422.820781] Key type ._llcrypt unregistered [ 4447.859900] Key type ._llcrypt registered [ 4447.866120] Key type .llcrypt registered [ 4447.967638] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4449.182842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4449.207742] alg: No test for adler32 (adler32-zlib) [ 4450.607624] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 4450.992308] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4452.696223] Key type lgssc registered [ 4454.026244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4462.592535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4462.631945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.123436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4468.450110] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4474.342289] LustreError: 142059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.370315] LustreError: 142059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4477.608975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4478.244844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4483.667330] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4491.669946] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4497.979659] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4504.580617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.875415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4510.018353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4511.157973] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4519.258241] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4523.321292] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4529.295236] Lustre: 144642:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4533.134892] Lustre: 144690:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4534.702640] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4535.778863] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4540.404599] Lustre: 144836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4541.613982] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4542.732711] Lustre: 144935:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4549.096721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4549.111409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.130511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4549.139017] Lustre: Skipped 1 previous similar message [ 4553.659684] Lustre: server umount lustre-OST0000 complete [ 4559.333195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4559.348829] Lustre: Skipped 1 previous similar message [ 4559.358588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4559.364987] Lustre: Skipped 1 previous similar message [ 4564.453482] LustreError: 142056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4564.476591] LustreError: 142056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4564.571335] Lustre: server umount lustre-MDT0000 complete [ 4569.050342] LustreError: 142038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574102 with bad export cookie 100180838081160417 [ 4569.052064] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4569.063317] LustreError: 142038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4569.798228] Lustre: server umount lustre-MDT0001 complete [ 4585.836329] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4589.325681] Key type lgssc unregistered [ 4589.768795] LNet: 146000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4589.792907] LNetError: 146000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4589.835471] LNet: Removed LNI 192.168.201.156@tcp [ 4591.005895] Key type .llcrypt unregistered [ 4591.007762] Key type ._llcrypt unregistered [ 4621.421595] Key type ._llcrypt registered [ 4621.423692] Key type .llcrypt registered [ 4621.529302] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4623.036889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4623.083425] alg: No test for adler32 (adler32-zlib) [ 4624.635318] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 4624.980782] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4626.704219] Key type lgssc registered [ 4628.503454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4637.770293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4637.804118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.396065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.891833] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4649.959121] LustreError: 147416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4649.977268] LustreError: 147416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4654.365334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4654.894387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4660.168351] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4670.520934] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4679.570641] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4688.227167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4688.631408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.392189] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4699.904309] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4707.054960] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4709.110254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4717.899898] Lustre: *** cfs_fail_loc=159, val=0*** [ 4733.719925] Lustre: lustre-MDT0000: Client d4587896-223f-4ab3-b927-042b936c09d5 (at 192.168.201.56@tcp) reconnecting [ 4739.532554] Lustre: server umount lustre-OST0000 complete [ 4745.189878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4745.192335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4745.204185] Lustre: Skipped 1 previous similar message [ 4745.222389] Lustre: Skipped 1 previous similar message [ 4750.306041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4750.318422] Lustre: Skipped 1 previous similar message [ 4754.693287] Lustre: server umount lustre-MDT0000 complete [ 4755.448690] LustreError: 147416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4755.472706] LustreError: 147416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4756.449489] Lustre: 146727:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574273/real 1788574273] req@0000000073700b28 x1875455935287296/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788574289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4756.487789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.515474] LustreError: 148406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4756.531526] LustreError: 148406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4757.537238] Lustre: 146730:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788574274/real 1788574274] req@00000000d53522c0 x1875455935288704/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788574290 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4759.973931] LustreError: 147397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574292 with bad export cookie 3594932469402146080 [ 4759.977243] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4759.988171] LustreError: 147397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4760.544396] Lustre: server umount lustre-MDT0001 complete [ 4774.894910] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4778.834328] Key type lgssc unregistered [ 4779.324081] LNet: 151087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4779.352176] LNetError: 151087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4780.402134] LNet: Removed LNI 192.168.201.156@tcp [ 4782.044646] Key type .llcrypt unregistered [ 4782.050583] Key type ._llcrypt unregistered [ 4815.060958] Key type ._llcrypt registered [ 4815.072533] Key type .llcrypt registered [ 4815.283381] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4816.912203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4817.132902] alg: No test for adler32 (adler32-zlib) [ 4818.342926] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 4818.647925] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4820.322083] Key type lgssc registered [ 4822.339480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4833.833860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4833.886945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.751675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.921432] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4845.028632] LustreError: 152503:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.043469] LustreError: 152503:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4850.151389] LustreError: 152502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.826708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4854.364072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4860.137328] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4869.638395] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4876.476897] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4885.596780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.082247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.197873] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4899.401578] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4905.547452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4906.454172] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4928.538977] Lustre: lustre-OST0000: evicting 1e741300-8103-4ce1-a9cd-02ae6e656617 (at 192.168.201.56@tcp) by administrative request [ 4928.559681] Lustre: Skipped 1 previous similar message [ 4960.738521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4960.764517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.777117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4961.762508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4961.766831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4961.769038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.793398] Lustre: Skipped 1 previous similar message [ 4963.853801] Lustre: server umount lustre-OST0000 complete [ 4972.004892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4972.008852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4972.022575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4972.035757] Lustre: Skipped 1 previous similar message [ 4972.052756] Lustre: Skipped 1 previous similar message [ 4975.471802] Lustre: server umount lustre-MDT0000 complete [ 4977.123244] LustreError: 153234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4977.149218] LustreError: 153234:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4980.078459] LustreError: 152483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574513 with bad export cookie 5476331484513734115 [ 4980.085155] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4980.096995] LustreError: 152483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4980.820570] Lustre: server umount lustre-MDT0001 complete [ 4996.250084] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5000.767268] Key type lgssc unregistered [ 5001.315619] LNet: 156583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5001.331630] LNetError: 156583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5002.345729] LNet: Removed LNI 192.168.201.156@tcp [ 5003.773726] Key type .llcrypt unregistered [ 5003.776233] Key type ._llcrypt unregistered [ 5102.509566] Key type ._llcrypt registered [ 5102.520188] Key type .llcrypt registered [ 5102.725707] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5104.175419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5104.667449] alg: No test for adler32 (adler32-zlib) [ 5106.002057] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 5106.438355] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5108.208986] Key type lgssc registered [ 5109.614711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5119.650421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5119.679784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5121.476093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5127.638730] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5131.761154] LustreError: 159076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.795725] LustreError: 159076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5136.872837] LustreError: 159077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5139.838310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5140.477341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5146.507057] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5156.659723] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5163.943943] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5174.008785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5174.783248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5182.758712] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5185.098655] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5191.259929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5193.624926] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5210.593926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5210.618314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.640642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5211.625696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5211.626433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.619373] Lustre: server umount lustre-OST0000 complete [ 5221.859243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5221.860841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5221.895720] Lustre: Skipped 1 previous similar message [ 5221.902737] Lustre: Skipped 1 previous similar message [ 5225.890882] Lustre: server umount lustre-MDT0000 complete [ 5226.986221] LustreError: 160012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5227.039443] LustreError: 160012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5230.857300] LustreError: 159058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574763 with bad export cookie 10372710961930877074 [ 5230.885855] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5231.373912] Lustre: server umount lustre-MDT0001 complete [ 5247.750875] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5251.295078] Key type lgssc unregistered [ 5251.701353] LNet: 162644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5251.742096] LNetError: 162644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5251.791597] LNet: Removed LNI 192.168.201.156@tcp [ 5253.060394] Key type .llcrypt unregistered [ 5253.063427] Key type ._llcrypt unregistered [ 5286.710709] Key type ._llcrypt registered [ 5286.712807] Key type .llcrypt registered [ 5286.862661] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5288.514496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5288.758329] alg: No test for adler32 (adler32-zlib) [ 5290.699259] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 5291.457161] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5293.354375] Key type lgssc registered [ 5295.539845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5317.514737] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 22:20:48 (1788574848) [ 5341.433786] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 5357.504810] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5408.011940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5408.067623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.606710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5409.696216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5409.825968] Lustre: lustre-MDT0000: new disk, initializing [ 5409.943449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5409.972540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5416.117700] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5428.262152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.526494] Lustre: 166931:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5428.677852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5428.688227] Lustre: Skipped 1 previous similar message [ 5428.840264] Lustre: lustre-MDT0001: new disk, initializing [ 5428.988707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5429.069259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5429.090253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5435.145804] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5446.783399] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5453.727623] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5463.608353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5464.073863] Lustre: lustre-OST0000: new disk, initializing [ 5464.077878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5464.084511] Lustre: 168900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5464.222837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5472.080249] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5473.899271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5473.905630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5474.020486] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5483.563543] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5499.890956] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5499.897579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.909258] LustreError: Skipped 1 previous similar message [ 5499.912071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5499.927209] Lustre: Skipped 1 previous similar message [ 5499.969388] Lustre: Skipped 1 previous similar message [ 5502.007830] Lustre: server umount lustre-OST0000 complete [ 5510.114086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.122665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5510.135970] Lustre: Skipped 1 previous similar message [ 5510.165575] Lustre: Skipped 1 previous similar message [ 5513.675290] Lustre: server umount lustre-MDT0000 complete [ 5517.852252] LustreError: 166922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575050 with bad export cookie 854685389321305344 [ 5517.859075] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5517.860418] LustreError: 166922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5518.219559] Lustre: server umount lustre-MDT0001 complete [ 5532.190505] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5536.029929] Key type lgssc unregistered [ 5536.348729] LNet: 170810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5536.353316] LNetError: 170810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5536.378862] LNet: Removed LNI 192.168.201.156@tcp [ 5537.670697] Key type .llcrypt unregistered [ 5537.675944] Key type ._llcrypt unregistered [ 5568.861910] Key type ._llcrypt registered [ 5568.864349] Key type .llcrypt registered [ 5569.056542] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5570.358173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5570.504286] alg: No test for adler32 (adler32-zlib) [ 5571.835443] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 5572.152856] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5573.908228] Key type lgssc registered [ 5575.343895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5586.106227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5586.127520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.616186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.459521] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5597.156423] LustreError: 172223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5597.185214] LustreError: 172223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5601.413358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5601.842274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5606.838113] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5615.371618] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5621.836895] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5630.831153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5631.262255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5632.319168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5639.190428] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5648.675724] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5656.237167] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5656.272985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.299525] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5656.309136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.327206] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5657.618261] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5657.640888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.657286] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5657.670436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.689982] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5658.954555] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5658.980477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5658.996632] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5659.010384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.033584] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5660.163745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.248741] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5661.254500] Lustre: Skipped 1 previous similar message [ 5661.273268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5661.282563] Lustre: Skipped 1 previous similar message [ 5661.287676] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5661.293462] Lustre: Skipped 1 previous similar message [ 5661.297734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.309564] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5661.311935] Lustre: Skipped 1 previous similar message [ 5662.460856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.594412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.838710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.932751] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5665.936800] Lustre: Skipped 3 previous similar messages [ 5665.956697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.963326] Lustre: Skipped 3 previous similar messages [ 5665.966480] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5665.969203] Lustre: Skipped 3 previous similar messages [ 5665.971693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.976940] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5665.984080] Lustre: Skipped 3 previous similar messages [ 5666.968252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.040777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.983144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.913694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.936372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.152244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.326098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.195418] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5674.209251] Lustre: Skipped 7 previous similar messages [ 5674.245594] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.258609] Lustre: Skipped 7 previous similar messages [ 5674.263933] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5674.268599] Lustre: Skipped 7 previous similar messages [ 5674.273185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.289891] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5674.298690] Lustre: Skipped 7 previous similar messages [ 5675.435249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.536189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.489989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.527448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.489265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.427688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.328489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.251624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.185606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.194079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.145697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.782520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.483046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.657797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.865265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.912121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.961297] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5690.967994] Lustre: Skipped 16 previous similar messages [ 5691.002182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.023893] Lustre: Skipped 16 previous similar messages [ 5691.044063] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5691.059523] Lustre: Skipped 16 previous similar messages [ 5691.066338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.100643] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5691.118272] Lustre: Skipped 16 previous similar messages [ 5692.336717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.230790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.317614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.644692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.902187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.261824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.246495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.687144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.892578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.432488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.659443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.721599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.685915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.740454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.842862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.863610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.093970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.009389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.051074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.062457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.213449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.618882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.643730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.762689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.359230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.241328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.116523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.375222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.613469] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5723.621773] Lustre: Skipped 28 previous similar messages [ 5723.652671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5723.662849] Lustre: Skipped 28 previous similar messages [ 5723.670877] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5723.676773] Lustre: Skipped 28 previous similar messages [ 5723.679299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.692987] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5723.702366] Lustre: Skipped 28 previous similar messages [ 5724.828248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.838516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.851308] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.989270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.988758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.962555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.217279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.206889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.178057] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.037606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.962182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.783909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.626118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.241124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.010939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.866588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.885264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.683468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.611528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.651087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.563521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.364353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.272266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.461484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.565957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.379181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.567387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.883925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.926073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.827771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.002222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.469061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.343299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.332159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.321655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.528818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.486830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.689955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.596085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.641773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.543453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.483796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.473469] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.683950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.801808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.839551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.257717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.235720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.209235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.096880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.029069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.995102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.060122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.086709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.015720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.138237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.069555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.043899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.998811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.982556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.968406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.907179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.705763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.789391] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5787.797544] Lustre: Skipped 63 previous similar messages [ 5787.823593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.834884] Lustre: Skipped 63 previous similar messages [ 5787.838761] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5787.842735] Lustre: Skipped 63 previous similar messages [ 5787.859746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.884364] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5787.892505] Lustre: Skipped 63 previous similar messages [ 5788.836074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.710547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.624661] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.619338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.704513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.621895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.849640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.034600] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.079560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.065640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.132242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.122583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.280091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.743497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.757687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.920201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.930849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.190255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.668534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.587314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.799893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.882359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.104244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.366670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.647706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.979612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.492533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.713193] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.000091] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.203230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.268074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.382927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.548137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.814315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.028585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.090605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.014044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.344138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.539884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.480847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.529632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.428173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.488086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.501992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.750512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.900096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.905319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.906817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.999739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.016819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.074241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.888230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.900136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.775683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5848.987882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.961663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.802086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.970559] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5853.871053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.871501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.061749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.311047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.333331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.292425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.423866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.458827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5862.309705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.201516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.130227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.129269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.887790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.729515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.510306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.493381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.430921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.510250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.526568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.616799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.770925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.974767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.007298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.318546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.524323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.496615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.664531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.554416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.479715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.431467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.891535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.830912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.795826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5888.096429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.346248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.405452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.506460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.647684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.600492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.561952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.796309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.844987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.964292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.068834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.982239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.235112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.502626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.462542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.734495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.865263] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5907.208635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.283236] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.548665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.199979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.866842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.840650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.651302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.701259] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5916.711425] Lustre: Skipped 115 previous similar messages [ 5916.753471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.786236] Lustre: Skipped 115 previous similar messages [ 5916.798288] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5916.814198] Lustre: Skipped 115 previous similar messages [ 5916.837311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.877424] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5916.889577] Lustre: Skipped 115 previous similar messages [ 5917.996464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.938205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.132974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.004996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.162743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.053500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.337238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.524609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.765182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.046291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.393341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.637307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.618597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.689790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.617279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.718524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5935.819841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.027822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.813285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.166657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.670682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.900140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.055802] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.344061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.348684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.423475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.281140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.241838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.198375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.571586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.816965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.952874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.067847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.144412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.093483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.084516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.271216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.255456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.321464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.456414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.576384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.832965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.722201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.832191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.679936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.749471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.743616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.976350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.948972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.140947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.226299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.162707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.337568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.190473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.959222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.650270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.609562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.512945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.594921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.410651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.140582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.880655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.800316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5987.767602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.368648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.292521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.378075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.333901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.429723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.284776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.243140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.217320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.304774] LustreError: 172243:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5996.324908] LustreError: 172243:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5996.330599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.383043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.380536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5999.087205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.088123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.966676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.870280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.830715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.896780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6004.840251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.797351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.863118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.860066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.832211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.729809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.169060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.383558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.424142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.380931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.245245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6016.278637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.133280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.140741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.970513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.989708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.863367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.735924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.676542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.754369] LustreError: 172243:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6023.778474] LustreError: 172243:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6023.782328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.878944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.740756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.692677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.741351] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6028.831982] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.986387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.214827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6032.367641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.549344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.813505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.808539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.948278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.030290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.952436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.850686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.806975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.819591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.883342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.898524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.907143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6045.920912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.074303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.281717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.416331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.499973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.540860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.769650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.939954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.938612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.222188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.189794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.316996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.073209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.231397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.422763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.316523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.289281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.049368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.988872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.999906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.814418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.868124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.706988] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.740896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.521681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.434533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.320772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.308113] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.162910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.085102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.915087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.832424] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.531902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.499428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.395333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.216517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.126264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.249814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.293425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.127229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.040837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.963201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.145786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.067303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.061260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6090.194068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.208451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.186297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.346110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.361202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.443042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.338222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.367940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.364491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.549270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.658220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.766591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.633991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.691665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.570170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.403579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.214767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.370770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.562897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.731892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.101610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.356215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.375079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.324910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.034204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.016561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.930221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.851798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6118.674166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.626765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.813069] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.890253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.822612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6123.751505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.598042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.738851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.605064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.468078] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.343577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.038391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.734242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.614536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.395666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.376192] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.113681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.946794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.700285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.426860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.215612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.139160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.892749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.755598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.747654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.653816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.130677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.098195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.234707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.646374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.826110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.923806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.948820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.231773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6151.509883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.502698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.338803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.121623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.949535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.940155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.046141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.928882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.223792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.226247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.212684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.073952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.007652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.849641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.830291] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.871294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.780581] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.652733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.599849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.472821] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.390147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.248646] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6173.252551] Lustre: Skipped 249 previous similar messages [ 6173.265841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.273296] Lustre: Skipped 249 previous similar messages [ 6173.275995] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6173.286192] Lustre: Skipped 249 previous similar messages [ 6173.292889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.304165] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6173.310690] Lustre: Skipped 249 previous similar messages [ 6174.140449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.895708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.728935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.629487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.632848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.620883] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.476400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.483506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.700593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.676304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.824578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.726727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.844603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.858654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.776620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.973659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.008101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.958745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.914396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.781817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.722736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.717292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.856131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.892242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.706810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.815359] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.672965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.015378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.989035] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.883403] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.595775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.484476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.332503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.055863] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.908824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.013552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.816671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.579310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.645303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.569375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.329793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.129817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.015992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.156132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.114345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.268954] LustreError: 172243:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6218.276890] LustreError: 172243:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6218.282402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.344531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.361705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.451913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.174333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.425741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6224.270332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.313701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.345118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.558697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.624425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.577112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.579343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.649634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.702990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.546379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.311825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.084209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.032692] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.838461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.741065] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.874394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.860409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.064461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.984701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.039948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.062871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.951690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.934941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.290492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.292347] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.447390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.777298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.784454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.570320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.415860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6254.292943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.064399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.965472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.117759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6257.653381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.391498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6259.243207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.341166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.513794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6262.712777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.657779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.765170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.607041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6266.768734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.906278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.234741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6270.214952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.364088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.407224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.364904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6274.730543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.883472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6277.108279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.168323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.290652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6280.475474] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.516741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6282.647749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.467086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.555400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6285.748784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.731349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.723977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.670753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.177068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.417415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.543674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6293.706410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.489161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.567676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.342913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.164800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.095780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6299.196648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.149223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.001020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.735823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.644621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.420504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.456394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.263618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.296456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6307.209247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.167198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.019729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.944675] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.719801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.677210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6312.468330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.281973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.998818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.868907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.954828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6316.819336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.645954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.573344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.708073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.750389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.748252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.632570] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.446299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.207854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.065489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.943854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.867796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.925392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.780275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.755840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.777435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.692401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.785465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.697877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.654329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.374135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.148256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.903130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.807860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6338.672902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.478697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.180991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.059273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.915733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.763141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.738870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.663252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.668536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.627794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.694895] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.589480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6349.394302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.240340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6351.246746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.284879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.241825] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.138848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6354.977877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.843439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.717621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.706811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.702817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.668940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.631501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.532552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.333333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.211636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.264531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.268073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.250893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.171268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.017538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.830426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.684393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.458334] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.286889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.089152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.959993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6373.778710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.464485] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.332583] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.193496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.994390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.859184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.018691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.884387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.973830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.637229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.608256] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.359230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.293294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.247122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.061320] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.796623] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.678924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6388.450137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.320022] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.974262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.723700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.625794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.457814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.363084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.224170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.966296] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.080482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.174814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.243131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.129343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.030117] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.891500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.761162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.561313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.512807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.261810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.098490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.955991] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.789703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.773694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.805492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.565439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.550355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.388838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6412.318502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.186312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.998651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.755348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.671413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.785319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.898226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.966790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.960496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.166431] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.167688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.132562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.090949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.000298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.811273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.680851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.710206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.633834] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.500924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.311456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.151635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.069339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.136796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.116846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.091729] LustreError: 172243:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6435.103743] LustreError: 172243:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 1 previous similar message [ 6435.114743] LustreError: 172243:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6435.126252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.128820] LustreError: 172243:0:(osp_precreate.c:1359:osp_precreate_thread()) Skipped 1 previous similar message [ 6436.097978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.102333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.167708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.073153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.963726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.739769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.645635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.579260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.146987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.325762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.563677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.416414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6448.284437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.106522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.043470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.982449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.978398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.960989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.142094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.246051] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.080935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.160549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.152981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.136922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.314522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.282440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.152702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.373144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.393144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.290674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.227650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.296965] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.163851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.136679] LustreError: 172243:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6470.150428] LustreError: 172243:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6470.159641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.087094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.943774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.836430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.989014] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.849252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.892398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.687776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.558096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.414496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.374634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.205902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.154245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.957974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.249747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.980385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.171915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.104161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6487.116817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.044593] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.994621] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.941254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.991223] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.027586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.068490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.979767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.255328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.068723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.113370] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.092443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.216810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.172734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.087826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.114852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.089527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.179422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.355118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.577278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.717499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.842459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.954739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.964765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.749251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.135337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.174442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.358544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.226817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.537222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.555993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.203856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.224484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.402609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.591931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.671961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.639171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.561524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6528.710754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.071089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.415882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.282478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.524411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.656287] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.677457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.960356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.111751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.965438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.937558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.906331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.251761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6545.099735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.156575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.406251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.760855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.645937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.661798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6552.622811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.561695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.392609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.564699] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.416854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.383501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.284371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6559.363311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.395738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.351264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.294245] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.589183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.393368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6565.624219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.546682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.592461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.540573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.874332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6571.029473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6572.224120] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6573.436527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.190838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.864564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6575.701644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.825925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6577.568164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6578.471849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6579.273713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.130898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.278354] LustreError: 172243:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6581.297838] LustreError: 172243:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6581.326284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.349174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.136171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.802222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6584.633814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6585.338484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6586.430854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.241145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6588.475659] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6590.015761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6590.993936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.148037] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.162488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6594.282844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6595.488216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6596.499224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.800194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.822868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6600.002330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6600.762611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6601.785446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6602.724138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6603.737123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6604.586567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6605.668670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6606.553759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6607.550369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6608.694888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.509445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6610.503052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6611.364396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6612.431319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6613.522136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6614.421500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6615.438878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6616.189188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6616.997798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6617.781325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6619.005789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6620.120683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6621.143282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6622.143259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6623.165728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6624.215537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6625.048202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6626.061127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6627.273983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6628.330800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6629.096269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6629.950437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6630.886741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6631.640371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6632.678350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6633.610419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6634.589443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6635.326608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6636.368658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6637.507173] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6638.568695] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6639.836616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6640.688869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6641.548880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6642.532981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6643.633151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6644.544400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6645.520716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6646.508265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6647.699620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6648.640268] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6649.542586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6650.511577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6651.290992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6652.130467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6652.925085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6653.631131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6654.366260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6655.240289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6656.136277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6657.016592] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6657.940909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6659.018367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6660.108119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6661.080459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6661.865729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6662.898711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6663.901780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6664.848196] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6665.710814] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6666.553247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6667.441310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6668.246233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6669.027362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6669.844730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6670.601452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6981.515324] LustreError: 174048:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6981.527723] LustreError: 172243:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7400.419970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7400.428681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.435413] Lustre: Skipped 507 previous similar messages [ 7400.439626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7404.009417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7404.027673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7405.030147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7405.048446] Lustre: Skipped 1 previous similar message [ 7406.279518] Lustre: server umount lustre-OST0000 complete [ 7409.127464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7409.142374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7415.268429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7415.279029] Lustre: Skipped 3 previous similar messages [ 7416.036218] Lustre: server umount lustre-MDT0000 complete [ 7420.387985] LustreError: 172223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7420.436646] LustreError: 172223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7420.787371] LustreError: 172205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576953 with bad export cookie 17326725279530682525 [ 7420.791157] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7420.809146] LustreError: 172205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7421.619583] Lustre: server umount lustre-MDT0001 complete [ 7432.751066] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7435.744344] Key type lgssc unregistered [ 7436.164612] LNet: 227444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7436.173586] LNetError: 227444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7436.213283] LNet: Removed LNI 192.168.201.156@tcp [ 7437.187323] Key type .llcrypt unregistered [ 7437.188844] Key type ._llcrypt unregistered [ 7462.338285] Key type ._llcrypt registered [ 7462.339754] Key type .llcrypt registered [ 7462.418628] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7463.796839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7463.814660] alg: No test for adler32 (adler32-zlib) [ 7464.875950] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 7465.096198] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7466.768213] Key type lgssc registered [ 7467.820954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7475.860626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7475.880583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.340667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.356758] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7487.458761] LustreError: 228862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7487.483149] LustreError: 228862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7491.232282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7491.775884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7496.392214] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7503.037875] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7507.665528] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7514.569774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7514.932324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7521.683543] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7522.109700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 7528.619302] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7534.324133] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7536.061880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7536.069726] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7536.073294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7536.084942] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7536.085138] LustreError: 228881:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7541.473776] Lustre: lustre-OST0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 7542.758325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.763625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7542.765265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7542.787925] Lustre: Skipped 1 previous similar message [ 7546.557869] Lustre: lustre-OST0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 7546.567101] Lustre: Skipped 1 previous similar message [ 7551.698164] Lustre: lustre-OST0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 7551.700678] Lustre: Skipped 2 previous similar messages [ 7551.968219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7552.135678] Lustre: server umount lustre-OST0000 complete [ 7556.160135] LustreError: 228881:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 7586.653266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7587.035409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7593.343465] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7598.327128] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7598.345770] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7598.364249] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7598.376547] Lustre: Skipped 1 previous similar message [ 7600.751889] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7608.766617] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 7613.650450] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ee15089d800.ost_server_uuid 50 [ 7614.754647] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ee15089d800.ost_server_uuid in FULL state after 0 sec [ 7619.047435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.060985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7624.528899] Lustre: server umount lustre-OST0000 complete [ 7629.281941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7629.299579] Lustre: Skipped 1 previous similar message [ 7629.306370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7629.311583] Lustre: Skipped 3 previous similar messages [ 7634.637775] Lustre: server umount lustre-MDT0000 complete [ 7637.982842] LustreError: 228840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577170 with bad export cookie 3620777845761442724 [ 7637.992414] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7638.007170] LustreError: 228840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7638.290941] Lustre: server umount lustre-MDT0001 complete [ 7649.531907] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7652.743636] Key type lgssc unregistered [ 7653.064707] LNet: 234072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7653.067627] LNetError: 234072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7653.099062] LNet: Removed LNI 192.168.201.156@tcp [ 7654.055778] Key type .llcrypt unregistered [ 7654.058624] Key type ._llcrypt unregistered [ 7689.121451] Key type ._llcrypt registered [ 7689.130154] Key type .llcrypt registered [ 7689.286696] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7690.979826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7691.012747] alg: No test for adler32 (adler32-zlib) [ 7692.128696] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 7692.429169] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7694.112196] Key type lgssc registered [ 7695.387239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7704.831579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7704.848578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7706.294932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7710.844427] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7716.847222] LustreError: 235802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7716.869686] LustreError: 235802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7719.918048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7720.356484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7725.399161] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7728.762776] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7735.606443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7736.017993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7742.617717] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7751.375629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 7752.612590] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7752.763959] Lustre: lustre-OST0001: new disk, initializing [ 7752.767193] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7752.772284] Lustre: 238124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7752.863464] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7759.510208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7759.530457] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7759.680568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7760.158743] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7771.280841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7781.694216] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7795.686086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.697753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7795.724147] Lustre: Skipped 3 previous similar messages [ 7795.727161] Lustre: Skipped 2 previous similar messages [ 7800.648765] Lustre: server umount lustre-MDT0000 complete [ 7800.828061] LustreError: 235803:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7800.850828] LustreError: 235803:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7804.744672] LustreError: 235781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577337 with bad export cookie 3114988892489109555 [ 7804.754527] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7804.761460] LustreError: 235781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7805.147595] Lustre: server umount lustre-MDT0001 complete [ 7819.579405] Lustre: server umount lustre-OST0000 complete [ 7821.480866] Lustre: 235067:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788577338/real 1788577338] req@ffff95bea3404700 x1875459152375552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788577354 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7821.544303] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.699222] Lustre: server umount lustre-OST0001 complete [ 7844.523934] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7858.288406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7858.607809] LustreError: 241231:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7858.662268] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7862.569222] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7863.787399] LustreError: 241232:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7868.908850] LustreError: 241231:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7872.047781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.443509] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7876.895720] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7880.088530] Lustre: 242373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7886.938308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.813107] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7897.521840] LustreError: 242727:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7902.713248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7902.941829] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7902.946385] Lustre: Skipped 1 previous similar message [ 7904.013062] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 7904.042762] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49760) [ 7909.205578] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7917.358515] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7920.731744] Lustre: 244243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7933.922886] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7933.930533] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.955205] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7933.961130] Lustre: Skipped 1 previous similar message [ 7934.975121] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7934.981131] Lustre: Skipped 3 previous similar messages [ 7937.977955] Lustre: server umount mylustre-MDT0000 complete [ 7940.069644] LustreError: 241231:0:(ldlm_lib.c:1192:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7940.085955] LustreError: 241231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7941.498889] LustreError: 241207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577474 with bad export cookie 3114988892489112362 [ 7941.505546] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7941.510825] LustreError: 241207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7941.777451] Lustre: server umount mylustre-MDT0001 complete [ 7955.660053] Lustre: server umount mylustre-OST0000 complete [ 7968.409102] Lustre: server umount mylustre-OST0001 complete [ 7990.879994] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8000.837029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.408173] LustreError: 246552:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8001.517592] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8006.550640] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8014.856760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.732225] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8024.778742] Lustre: 247694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8031.515147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8037.574923] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8045.142158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8047.416172] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49792) [ 8047.416592] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 8051.394679] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8058.723901] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8061.655330] Lustre: 249567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8070.614109] Lustre: server umount tfs-MDT0000 complete [ 8071.653911] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8071.659474] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8071.676504] Lustre: Skipped 3 previous similar messages [ 8071.683701] LustreError: 246552:0:(ldlm_lib.c:1192:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8071.697842] LustreError: 246552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8073.999920] LustreError: 246528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577607 with bad export cookie 3114988892489114252 [ 8074.014147] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8074.017389] LustreError: 246528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8074.289459] Lustre: server umount tfs-MDT0001 complete [ 8088.637362] Lustre: server umount tfs-OST0000 complete [ 8103.142932] Lustre: server umount tfs-OST0001 complete [ 8124.075604] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8132.918047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.418775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8133.426046] Lustre: Skipped 3 previous similar messages [ 8137.735577] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8144.632449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8149.065717] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8151.670381] Lustre: 252958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8158.153608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8164.888611] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8173.349256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8179.718106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8179.728529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49730 to 0x280000401:49824) [ 8180.203793] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8187.194558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8190.906190] Lustre: 254828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8197.242068] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 23:08:49 (1788577729) [ 8215.533805] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8215.556807] Lustre: Skipped 4 previous similar messages [ 8215.556931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8215.583063] Lustre: Skipped 3 previous similar messages [ 8218.884948] Lustre: server umount lustre-MDT0000 complete [ 8220.642523] LustreError: 252550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8220.679904] LustreError: 252550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8222.539281] LustreError: 251792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577755 with bad export cookie 3114988892489115918 [ 8222.550043] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8222.555282] LustreError: 251792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8225.769485] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8225.771907] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8225.780854] Lustre: Skipped 3 previous similar messages [ 8225.796899] Lustre: Skipped 1 previous similar message [ 8228.923644] Lustre: server umount lustre-MDT0001 complete [ 8247.267552] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8247.634580] Lustre: server umount lustre-OST0000 complete [ 8257.619292] Lustre: server umount lustre-OST0001 complete [ 8264.845400] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 8275.325609] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8308.686683] LDISKFS-fs (dm-0): Mount option "noacl" will be removed by 3.5 [ 8308.686683] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8308.686683] [ 8308.711831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8308.879746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8308.898539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8308.975632] Lustre: lustre-MDT0000: new disk, initializing [ 8309.078271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8309.082909] Lustre: Skipped 3 previous similar messages [ 8309.099957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8312.536655] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8321.053997] LDISKFS-fs (dm-1): Mount option "noacl" will be removed by 3.5 [ 8321.053997] Contact linux-ldiskfs@vger.kernel.org if you think we should keep it. [ 8321.053997] [ 8321.080918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,errors=remount-ro,noacl,no_mbcache,nodelalloc [ 8321.193330] Lustre: 258011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8321.245978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8321.249826] Lustre: Skipped 1 previous similar message [ 8321.310764] Lustre: lustre-MDT0001: new disk, initializing [ 8321.424227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8321.434711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8325.111533] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8334.251580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8334.565229] Lustre: lustre-OST0000: new disk, initializing [ 8334.569423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8334.575867] Lustre: 259592:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8336.215892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8336.230531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8336.388301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8339.864579] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8348.499377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8348.731324] Lustre: lustre-OST0001: new disk, initializing [ 8348.736667] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8348.753067] Lustre: 260463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8350.576850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8350.587384] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8350.687193] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8354.426463] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8366.532080] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 23:11:38 (1788577898) [ 8380.893738] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 23:11:53 (1788577913) [ 8396.777893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8396.787184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8396.791154] Lustre: Skipped 3 previous similar messages [ 8401.184840] Lustre: server umount lustre-OST0000 complete [ 8411.830292] Lustre: server umount lustre-MDT0000 complete [ 8415.410040] LustreError: 260462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577948 with bad export cookie 3114988892489117472 [ 8415.412895] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8415.417735] LustreError: 260462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8417.251369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8417.260252] Lustre: Skipped 3 previous similar messages [ 8421.879689] Lustre: server umount lustre-MDT0001 complete [ 8433.127189] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8510.926498] Lustre: server umount lustre-OST0001 complete [ 8518.304508] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 8535.535549] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8573.277878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8573.624294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8573.659843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8573.740900] Lustre: lustre-MDT0000: new disk, initializing [ 8573.820913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8573.824560] Lustre: Skipped 3 previous similar messages [ 8573.843889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8578.825173] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8589.974430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.128704] Lustre: 265265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8590.186456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8590.188636] Lustre: Skipped 1 previous similar message [ 8590.261617] Lustre: lustre-MDT0001: new disk, initializing [ 8590.386424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8590.401230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8595.326250] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8606.556085] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8613.133780] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8620.575407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8620.954649] Lustre: 267233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8622.119041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8622.219104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8626.700506] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8635.469633] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8642.538541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8642.544690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8642.556919] Lustre: Skipped 5 previous similar messages [ 8642.561858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8642.567960] Lustre: Skipped 1 previous similar message [ 8648.490054] Lustre: server umount lustre-OST0000 complete [ 8651.749898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8651.760912] LustreError: Skipped 1 previous similar message [ 8657.799739] Lustre: server umount lustre-MDT0000 complete [ 8658.402836] LustreError: 265276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8658.423730] LustreError: 265276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 8662.625906] LustreError: 265256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578195 with bad export cookie 3114988892489118942 [ 8662.629149] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8662.639855] LustreError: 265256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8663.158860] Lustre: server umount lustre-MDT0001 complete [ 8675.621271] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8679.159347] Key type lgssc unregistered [ 8679.682663] LNet: 268993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8679.690878] LNetError: 268993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8680.742824] LNet: Removed LNI 192.168.201.156@tcp [ 8682.037265] Key type .llcrypt unregistered [ 8682.041798] Key type ._llcrypt unregistered [ 8709.186315] Key type ._llcrypt registered [ 8709.189262] Key type .llcrypt registered [ 8709.268743] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8710.328643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8710.374221] alg: No test for adler32 (adler32-zlib) [ 8711.531433] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 8711.773127] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 8713.440494] Key type lgssc registered [ 8714.967383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8723.096420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8723.119840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8724.591074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.154339] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8734.187348] LustreError: 270461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8734.218757] LustreError: 270461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8737.878437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.192518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8742.595704] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8751.720496] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8757.817976] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8765.354310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8765.888916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.016342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8773.006593] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8781.156820] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8782.139335] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8785.377448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8785.387581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.404898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8788.459378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8788.468912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.471231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8788.489776] Lustre: Skipped 1 previous similar message [ 8790.634685] Lustre: server umount lustre-OST0000 complete [ 8794.593738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8794.599326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.616995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8798.710758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8798.712110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8800.748911] Lustre: server umount lustre-MDT0000 complete [ 8803.810162] LustreError: 270456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8803.828901] LustreError: 270456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8805.243784] LustreError: 270439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578338 with bad export cookie 17519931571645207324 [ 8805.271733] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8805.273943] LustreError: 270439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8805.889056] Lustre: server umount lustre-MDT0001 complete [ 8818.244206] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8821.720213] Key type lgssc unregistered [ 8822.064799] LNet: 274047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8822.072991] LNetError: 274047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8822.090662] LNet: Removed LNI 192.168.201.156@tcp [ 8822.916886] Key type .llcrypt unregistered [ 8822.918590] Key type ._llcrypt unregistered [ 8840.845050] Key type ._llcrypt registered [ 8840.847965] Key type .llcrypt registered [ 8840.967723] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8842.224107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8842.294206] alg: No test for adler32 (adler32-zlib) [ 8843.337268] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 8843.585952] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 8845.272929] Key type lgssc registered [ 8846.609496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8856.537143] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8856.563886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8872.096493] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8877.216355] LustreError: 275242:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 8903.968227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.763494] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8919.146462] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8921.300411] Lustre: server umount lustre-OST0000 complete [ 8939.440984] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8942.211398] Key type lgssc unregistered [ 8942.487669] LNet: 276875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8942.491694] LNetError: 276875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8942.510169] LNet: Removed LNI 192.168.201.156@tcp [ 8943.289636] Key type .llcrypt unregistered [ 8943.291513] Key type ._llcrypt unregistered [ 8985.547687] Key type ._llcrypt registered [ 8985.553727] Key type .llcrypt registered [ 8985.694329] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8987.247655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8987.302835] alg: No test for adler32 (adler32-zlib) [ 8988.464420] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 8988.873742] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 8990.640690] Key type lgssc registered [ 8991.864459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9005.142437] loop0: detected capacity change from 0 to 204800000 [ 9005.400628] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9005.420785] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9005.961389] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9007.117482] loop0: detected capacity change from 0 to 204800000 [ 9007.331445] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9007.342705] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9007.888421] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9009.298227] loop0: detected capacity change from 0 to 204800000 [ 9009.486476] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9009.505800] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9009.652623] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9011.149252] loop0: detected capacity change from 0 to 204800000 [ 9011.488373] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9011.517557] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9011.619418] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9012.681215] loop0: detected capacity change from 0 to 204800000 [ 9012.715024] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9017.141474] loop0: detected capacity change from 0 to 204800000 [ 9017.191931] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9020.968377] loop0: detected capacity change from 0 to 204800000 [ 9020.990625] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9025.226224] loop0: detected capacity change from 0 to 204800000 [ 9025.270876] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9029.649702] loop0: detected capacity change from 0 to 204800000 [ 9029.736129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9029.797202] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9029.882643] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9029.889516] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9029.898796] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9029.904521] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9029.914384] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9029.929586] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9029.933863] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9029.940924] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9029.946953] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9029.953704] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9029.960967] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9029.967608] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9029.974733] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9029.985707] Lustre: 280289:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9035.901214] Lustre: server umount MGS complete [ 9036.829669] loop0: detected capacity change from 0 to 204800000 [ 9036.923502] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9037.422255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9037.517555] Lustre: lustre-MDT0000: Aborting client recovery [ 9038.813099] loop1: detected capacity change from 0 to 204800000 [ 9038.947386] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9039.052868] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9039.068807] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9039.079243] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9039.107321] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9039.117792] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9039.128150] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9039.139511] Lustre: 280723:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9039.161290] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9039.307066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9039.354882] Lustre: lustre-MDT0001: Aborting client recovery [ 9040.404097] loop2: detected capacity change from 0 to 204800000 [ 9040.618041] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9040.707417] Lustre: 280802:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9040.736218] Lustre: 280802:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9040.767451] Lustre: 280802:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9040.792488] Lustre: 280802:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9040.808562] Lustre: 280802:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9040.854429] Lustre: 280802:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9040.887709] Lustre: 280802:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9041.063430] Lustre: Found index 0 for lustre-OST0000, updating log [ 9041.159360] Lustre: 280821:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9041.285304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9041.338753] Lustre: lustre-OST0000: Aborting recovery [ 9042.676330] loop3: detected capacity change from 0 to 204800000 [ 9042.895956] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9043.019326] Lustre: 280887:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9043.033549] Lustre: 280887:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9043.047125] Lustre: 280887:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9043.054568] Lustre: 280887:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9043.067136] Lustre: 280887:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9043.074923] Lustre: 280887:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9043.082415] Lustre: 280887:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9043.102825] Lustre: Found index 1 for lustre-OST0001, updating log [ 9043.140308] Lustre: 280892:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9043.230787] Lustre: lustre-OST0001: Aborting recovery [ 9047.339130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9047.350720] LustreError: 280666:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9047.380629] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9047.394553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9047.432309] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9049.403265] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9049.412199] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9052.760609] LustreError: 280633:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 198us: rc = -2 [ 9053.268137] LustreError: 280632:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 201us: rc = -2 [ 9053.280691] LustreError: 280632:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 16 previous similar messages [ 9070.053485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9070.068560] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9070.078845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9070.120645] Lustre: Skipped 3 previous similar messages [ 9072.508670] Lustre: server umount lustre-MDT0000 complete [ 9073.338320] LustreError: 280615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578606 with bad export cookie 9535541525410990922 [ 9073.360355] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9075.169168] LustreError: 280630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9075.171513] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9075.191709] LustreError: 280630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9075.213584] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9075.217741] Lustre: Skipped 4 previous similar messages [ 9078.177445] LustreError: 280632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9078.182297] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9078.191131] LustreError: 280632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9078.204472] Lustre: Skipped 1 previous similar message [ 9080.136874] Lustre: server umount lustre-MDT0001 complete [ 9087.861361] Lustre: server umount lustre-OST0000 complete [ 9095.176508] Lustre: server umount lustre-OST0001 complete [ 9103.849033] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 23:23:56 (1788578636) [ 9134.642881] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 9146.299745] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9185.296483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9185.625919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9185.686415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9185.747497] Lustre: lustre-MDT0000: new disk, initializing [ 9185.845606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9185.852348] Lustre: Skipped 1 previous similar message [ 9185.866561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9190.843205] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9202.284866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.469818] Lustre: 284958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9202.536913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9202.541650] Lustre: Skipped 1 previous similar message [ 9202.597415] Lustre: lustre-MDT0001: new disk, initializing [ 9202.677385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9202.710534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9202.726350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9207.633331] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9218.457800] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9225.287345] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9232.891572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9233.168954] Lustre: lustre-OST0000: new disk, initializing [ 9233.172737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9233.178696] Lustre: 286930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9233.269459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.664760] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9240.690787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9240.699378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9240.853990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9249.838717] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9255.568881] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9266.876612] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9274.600629] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9286.447964] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 9333.221734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9333.223480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9333.230624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9333.246407] Lustre: Skipped 2 previous similar messages [ 9335.018826] Lustre: server umount lustre-OST0000 complete [ 9340.901361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9340.909392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9340.923862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9340.926295] Lustre: Skipped 1 previous similar message [ 9345.023310] Lustre: server umount lustre-MDT0000 complete [ 9348.577374] LustreError: 284970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9348.586733] LustreError: 284970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9348.606388] LustreError: 284950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578881 with bad export cookie 9535541525410993638 [ 9348.611195] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9348.612654] LustreError: 284950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9348.908241] Lustre: server umount lustre-MDT0001 complete [ 9359.618056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9364.769601] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9370.687498] Lustre: server umount MGS complete [ 9382.771878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9383.133133] LustreError: 290872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9383.162632] LustreError: 290885:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9383.218101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9388.518477] LustreError: 290873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9388.667810] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9397.995182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.243717] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9412.886410] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9419.472243] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9426.941406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9427.381206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9427.389880] Lustre: Skipped 1 previous similar message [ 9434.059167] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9440.720960] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9468.552867] Lustre: server umount lustre-OST0000 complete [ 9475.051829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9475.059085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9475.077315] Lustre: Skipped 1 previous similar message [ 9475.085747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9475.090643] Lustre: Skipped 2 previous similar messages [ 9482.917550] Lustre: server umount lustre-MDT0000 complete [ 9485.793893] LustreError: 292327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9485.824848] LustreError: 292327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9486.817990] Lustre: 278256:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788579003/real 1788579003] req@000000009072db64 x1875460511427584/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788579019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9486.818127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9486.837780] Lustre: 278256:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9486.859743] Lustre: Skipped 1 previous similar message [ 9487.132533] LustreError: 290851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579020 with bad export cookie 9535541525410995591 [ 9487.155427] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9487.464505] Lustre: server umount lustre-MDT0001 complete [ 9500.766675] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 9504.106668] Key type lgssc unregistered [ 9504.627061] LNet: 294818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9504.642486] LNetError: 294818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9504.688573] LNet: Removed LNI 192.168.201.156@tcp [ 9505.681177] Key type .llcrypt unregistered [ 9505.684384] Key type ._llcrypt unregistered [ 9545.978827] Key type ._llcrypt registered [ 9545.980236] Key type .llcrypt registered [ 9546.104810] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 9561.343575] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9562.538313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9562.588688] alg: No test for adler32 (adler32-zlib) [ 9563.682756] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 9564.043501] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 9565.743245] Key type lgssc registered [ 9567.254608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9603.932194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9603.965763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.276677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9605.307289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9605.427661] Lustre: lustre-MDT0000: new disk, initializing [ 9605.500719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9605.539361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9610.075901] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9621.426981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.569058] Lustre: 298778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9621.659425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9621.669814] Lustre: Skipped 1 previous similar message [ 9621.772777] Lustre: lustre-MDT0001: new disk, initializing [ 9621.869399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9621.906954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9621.922931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9626.825168] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9635.863657] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9641.697441] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9647.684764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9647.945976] Lustre: lustre-OST0000: new disk, initializing [ 9647.951556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9647.956530] Lustre: 300749:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9648.043700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9653.127165] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9656.383634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9656.394700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9656.510540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9662.120235] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9668.286403] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9678.214758] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9686.540670] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9738.724257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9738.733163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9738.751170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9741.679241] Lustre: server umount lustre-OST0000 complete [ 9748.964224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.968184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9748.991765] Lustre: Skipped 1 previous similar message [ 9748.995449] Lustre: Skipped 2 previous similar messages [ 9752.315325] Lustre: server umount lustre-MDT0000 complete [ 9756.348596] LustreError: 298768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579289 with bad export cookie 16376883261360917934 [ 9756.354228] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9756.364257] LustreError: 298768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9756.773558] Lustre: server umount lustre-MDT0001 complete [ 9767.090187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.113706] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9779.108640] Lustre: server umount MGS complete [ 9792.098565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9792.621150] LustreError: 304613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.645569] LustreError: 304613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9792.659677] LustreError: 304626:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9792.750875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.000658] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9798.117685] LustreError: 304614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9802.221060] LustreError: 304613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9807.333500] LustreError: 304614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9807.408567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9807.848915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9813.829620] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9822.891698] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9830.330591] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9837.674811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9838.123236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9845.514901] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9855.096388] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9898.991289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9898.996930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9899.002672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9899.002681] Lustre: Skipped 1 previous similar message [ 9899.008487] Lustre: Skipped 1 previous similar message [ 9899.034541] LustreError: Skipped 2 previous similar messages [ 9901.718060] Lustre: server umount lustre-OST0000 complete [ 9909.220469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9909.228993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9909.252150] Lustre: Skipped 1 previous similar message [ 9909.282678] Lustre: Skipped 2 previous similar messages [ 9912.775967] Lustre: server umount lustre-MDT0000 complete [ 9914.349331] LustreError: 304613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.359976] LustreError: 304613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9917.168935] LustreError: 304595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579450 with bad export cookie 16376883261360919775 [ 9917.174489] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9917.187101] LustreError: 304595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9917.697975] Lustre: server umount lustre-MDT0001 complete [ 9930.445318] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 9933.590929] Key type lgssc unregistered [ 9933.977845] LNet: 308586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9933.986878] LNetError: 308586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9934.006891] LNet: Removed LNI 192.168.201.156@tcp [ 9935.208421] Key type .llcrypt unregistered [ 9935.210058] Key type ._llcrypt unregistered [ 9958.795508] Key type ._llcrypt registered [ 9958.797539] Key type .llcrypt registered [ 9958.884951] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 9974.106722] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9975.749397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9975.903775] alg: No test for adler32 (adler32-zlib) [ 9977.434290] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [ 9977.839731] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 9979.712240] Key type lgssc registered [ 9981.134128] Lustre: Echo OBD driver; http://www.lustre.org/ [10005.485374] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10020.113772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10020.176892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10021.518387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10021.595320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10021.748283] Lustre: lustre-MDT0000: new disk, initializing [10021.853140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10021.864467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10026.136237] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10035.894202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10036.138585] Lustre: lustre-OST0000: new disk, initializing [10036.143280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10036.152256] Lustre: Skipped 1 previous similar message [10036.162176] Lustre: 312629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10036.277498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10038.516843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10038.525950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10038.559633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10042.852214] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10823.685882] Lustre: server umount lustre-MDT0000 complete [10827.788960] LustreError: 311814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580360 with bad export cookie 16890762554852658554 [10827.808784] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10838.284027] Lustre: server umount lustre-OST0000 complete [10869.352451] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [10879.527817] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10916.165551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10916.492677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10916.515163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10916.595401] Lustre: lustre-MDT0000: new disk, initializing [10916.697416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10916.719386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10921.132416] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10932.992359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10933.181993] Lustre: 316957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10933.255909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10933.259739] Lustre: Skipped 1 previous similar message [10933.398668] Lustre: lustre-MDT0001: new disk, initializing [10933.529296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10933.580058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10933.605660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10939.395788] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10950.363825] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10957.744849] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10965.352842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10965.684775] Lustre: lustre-OST0000: new disk, initializing [10965.692725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10965.701731] Lustre: 318926:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10965.799559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10972.291263] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10973.790175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10973.808482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10973.904668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10981.855434] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10987.495814] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10987.802652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10993.003735] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10993.204427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10999.778612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10999.781338] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10999.813568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10999.823415] Lustre: Skipped 1 previous similar message [11001.502461] Lustre: server umount lustre-OST0000 complete [11010.019929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11010.044919] Lustre: Skipped 1 previous similar message [11010.056747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11010.063142] Lustre: Skipped 1 previous similar message [11015.143237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.150883] Lustre: Skipped 1 previous similar message [11019.235441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11019.245950] Lustre: Skipped 1 previous similar message [11020.256149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11020.525984] Lustre: server umount lustre-MDT0000 complete [11024.402391] LustreError: 316949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580557 with bad export cookie 16890762554855599373 [11024.411863] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11024.936576] Lustre: server umount lustre-MDT0001 complete [11028.674418] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11030.442296] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 23:56:02 (1788580562) [11037.257130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11037.733375] LustreError: 321238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11037.751215] LustreError: 321238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11037.859617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11042.857339] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11043.305366] LustreError: 321239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11047.394471] LustreError: 321238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11052.527110] LustreError: 321239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11052.868706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11053.435380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11058.211652] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11065.679939] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11071.502415] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11078.807763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11079.182874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11085.751494] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11093.485784] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11100.909920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11101.172815] Lustre: lustre-OST0001: new disk, initializing [11101.184083] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11101.194966] Lustre: 324073:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11101.362374] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11107.475484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11107.489729] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11107.549954] LustreError: 324401:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11107.881169] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11108.578902] LustreError: 324401:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11108.586924] LustreError: 324401:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11109.608987] LustreError: 324401:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11109.633866] LustreError: 324401:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11111.716446] LustreError: 324401:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11111.735988] LustreError: 324401:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 4 previous similar messages [11115.812147] LustreError: 324391:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11115.833425] LustreError: 324391:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 6 previous similar messages [11118.479503] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11118.712956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11123.569611] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9ee18084b800.ost_server_uuid 50 [11124.076538] LustreError: 324401:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11124.093686] LustreError: 324401:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [11125.075639] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ee18084b800.ost_server_uuid in FULL state after 0 sec [11130.704703] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ee18084b800.ost_server_uuid 50 [11132.222892] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ee18084b800.ost_server_uuid in FULL state after 0 sec [11134.390596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11148.777437] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11148.792288] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11148.800772] Lustre: Skipped 2 previous similar messages [11148.801208] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11148.840913] Lustre: Skipped 1 previous similar message [11153.909867] LustreError: 323061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.198410] Lustre: server umount lustre-OST0001 complete [11164.131335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11164.132145] LustreError: 323060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11164.136513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11164.137047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11164.137051] Lustre: Skipped 1 previous similar message [11164.150769] Lustre: Skipped 1 previous similar message [11164.189240] LustreError: 323060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [11169.181652] Lustre: server umount lustre-OST0000 complete [11176.432195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11176.453427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11181.249720] Lustre: server umount lustre-MDT0000 complete [11184.612772] LustreError: 321238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11184.642986] LustreError: 321238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11186.387734] LustreError: 321219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580719 with bad export cookie 16890762554855600255 [11186.395763] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11186.405691] LustreError: 321219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11187.031984] Lustre: server umount lustre-MDT0001 complete [11199.857515] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11203.188144] Key type lgssc unregistered [11203.527251] LNet: 326769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11203.540926] LNetError: 326769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11204.583635] LNet: Removed LNI 192.168.201.156@tcp [11205.573544] Key type .llcrypt unregistered [11205.576282] Key type ._llcrypt unregistered [11232.874440] Key type ._llcrypt registered [11232.879258] Key type .llcrypt registered [11233.031717] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11234.451939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11234.502049] alg: No test for adler32 (adler32-zlib) [11235.590910] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [11235.917060] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11237.640283] Key type lgssc registered [11238.971322] Lustre: Echo OBD driver; http://www.lustre.org/ [11245.914885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11245.954772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11247.798949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11253.398571] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11257.321525] LustreError: 328193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11257.357942] LustreError: 328193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11261.959985] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11262.438787] LustreError: 328194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11267.566506] LustreError: 328193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11269.883607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11270.382473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11275.317780] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11285.638739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11285.957421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11292.074860] LustreError: 329877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11292.089875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [11293.136802] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11300.324850] LustreError: 329876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11300.337244] LustreError: 329876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11301.868786] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11309.215681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11309.393833] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11314.682428] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [11316.132828] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11323.988405] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11331.454236] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [11331.668273] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11437.541820] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.560454] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11437.565305] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11438.939340] Lustre: server umount lustre-OST0001 complete [11442.680821] LustreError: 329878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11442.711559] LustreError: 329878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11445.733996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11445.739845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11445.748125] Lustre: Skipped 1 previous similar message [11445.758525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11445.760988] Lustre: Skipped 1 previous similar message [11447.783818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.793703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11451.709991] Lustre: server umount lustre-OST0000 complete [11458.017176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11458.020107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11458.025531] Lustre: Skipped 1 previous similar message [11458.028974] Lustre: Skipped 1 previous similar message [11461.913465] Lustre: server umount lustre-MDT0000 complete [11463.141661] LustreError: 328194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11463.153327] LustreError: 328194:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11466.432471] LustreError: 328174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580999 with bad export cookie 8313210105506290965 [11466.433210] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11466.445586] LustreError: 328174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11467.250327] Lustre: server umount lustre-MDT0001 complete [11482.605146] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11487.547489] Key type lgssc unregistered [11488.022129] LNet: 333156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11488.030874] LNetError: 333156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11488.046618] LNet: Removed LNI 192.168.201.156@tcp [11488.916150] Key type .llcrypt unregistered [11488.919635] Key type ._llcrypt unregistered [11538.617924] Key type ._llcrypt registered [11538.627094] Key type .llcrypt registered [11538.844506] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11540.932996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11541.138689] alg: No test for adler32 (adler32-zlib) [11542.574884] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [11542.826615] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11544.528187] Key type lgssc registered [11546.375932] Lustre: Echo OBD driver; http://www.lustre.org/ [11556.721981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11556.761286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11558.369583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11563.828930] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11568.616312] LustreError: 335447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11568.628447] LustreError: 335447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11573.013114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11573.608405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11579.251565] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11582.489923] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11590.038035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11590.537309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11597.506590] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11598.706158] LustreError: 336935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.833416] LustreError: 336937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.867860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [11603.873776] LustreError: 336937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11607.563268] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11607.814640] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11613.226777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [11614.660605] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11624.505779] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11632.081860] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11649.000899] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.008571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11649.036238] Lustre: Skipped 3 previous similar messages [11649.052987] Lustre: Skipped 3 previous similar messages [11653.177462] Lustre: server umount lustre-MDT0000 complete [11654.120883] LustreError: 335444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11654.153708] LustreError: 335444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11657.156910] LustreError: 338456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581190 with bad export cookie 8907887316401169908 [11657.170641] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11657.177832] LustreError: 338456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11657.767563] Lustre: server umount lustre-MDT0001 complete [11673.046574] Lustre: server umount lustre-OST0000 complete [11675.937044] Lustre: 334712:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788581192/real 1788581192] req@ffff95bfbcc38700 x1875463189365888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788581208 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11675.967268] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11677.344329] Lustre: server umount lustre-OST0001 complete [11697.184588] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11706.958380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11707.324591] LustreError: 340713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11707.432471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11712.928641] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11717.609651] LustreError: 340714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11717.619577] LustreError: 340714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11723.401178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11724.214571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11730.931435] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11733.673153] Lustre: 341855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11739.957900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11740.324458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11746.952083] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11749.559272] LustreError: 342210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11749.586776] LustreError: 342210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11752.648143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [11756.632335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11762.174889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [11762.724187] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11769.808921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11773.181465] Lustre: 343727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11777.510300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11777.525451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11777.544955] Lustre: Skipped 1 previous similar message [11777.561538] Lustre: Skipped 2 previous similar messages [11782.627169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11782.636674] Lustre: Skipped 1 previous similar message [11783.571283] Lustre: server umount lustre-MDT0000 complete [11787.028533] LustreError: 341456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581320 with bad export cookie 8907887316401171609 [11787.045674] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11787.374763] Lustre: server umount lustre-MDT0001 complete [11801.395701] Lustre: server umount lustre-OST0000 complete [11803.107958] Lustre: 334712:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788581320/real 1788581320] req@ffff95bfc0eebb80 x1875463189415168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788581336 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11803.147225] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.166442] Lustre: Skipped 2 previous similar messages [11805.853933] Lustre: server umount lustre-OST0001 complete [11809.737684] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11811.409316] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11813.180287] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 00:09:05 (1788581345) [11836.927120] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11847.524223] loop0: detected capacity change from 0 to 1099511627776 [11848.230570] SGI XFS with ACLs, security attributes, quota, no debug enabled [11848.240876] *** VALIDATE xfs *** [11848.256892] XFS (loop0): Mounting V5 Filesystem [11848.297943] XFS (loop0): Ending clean mount [11849.510151] loop1: detected capacity change from 0 to 18691697672192 [11975.447698] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [11987.813280] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [12026.632536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12026.956467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12026.990353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12027.109434] Lustre: lustre-MDT0000: new disk, initializing [12027.176085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12027.179021] Lustre: Skipped 1 previous similar message [12027.197522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12031.929939] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12043.095226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12043.244388] Lustre: 348863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12043.296880] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12043.301822] Lustre: Skipped 1 previous similar message [12043.369987] Lustre: lustre-MDT0001: new disk, initializing [12043.513407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12043.536436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12047.953808] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12056.240503] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12061.162923] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12070.143291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12070.406816] Lustre: lustre-OST0000: new disk, initializing [12070.409966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12070.416048] Lustre: 350828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12072.142382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12072.180169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12072.378250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12078.329255] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12089.114831] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12094.613224] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12094.950751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12099.911610] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12100.145822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12103.139574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12103.146522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12103.154086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12103.158753] Lustre: Skipped 1 previous similar message [12103.169978] Lustre: Skipped 1 previous similar message [12108.262822] LustreError: 350810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12108.271459] LustreError: 350810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12108.347801] Lustre: server umount lustre-OST0000 complete [12113.388857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.400367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12113.406621] Lustre: Skipped 1 previous similar message [12113.429715] Lustre: Skipped 1 previous similar message [12118.094600] Lustre: server umount lustre-MDT0000 complete [12121.638730] LustreError: 348853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581654 with bad export cookie 8907887316401173156 [12121.653015] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12121.654029] LustreError: 348853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12122.159532] Lustre: server umount lustre-MDT0001 complete [12131.685658] XFS (loop0): Unmounting Filesystem [12138.317383] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 00:14:30 (1788581670) [12144.956497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12145.761503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12145.763930] Lustre: Skipped 2 previous similar messages [12149.484660] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12157.462337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12162.933765] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12171.524168] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12179.005109] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12186.248466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12193.740945] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12200.663698] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12213.219202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12213.225797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12213.234760] LustreError: Skipped 1 previous similar message [12213.236192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12213.278051] Lustre: Skipped 1 previous similar message [12218.344907] LustreError: 355213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12218.361269] LustreError: 355213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12218.446333] Lustre: server umount lustre-OST0000 complete [12229.031940] Lustre: server umount lustre-MDT0000 complete [12233.713271] LustreError: 353374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788581766 with bad export cookie 8907887316401174038 [12233.735986] LustreError: 353374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12233.737973] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12234.250974] Lustre: server umount lustre-MDT0001 complete [12245.061566] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12249.096263] Key type lgssc unregistered [12249.483533] LNet: 357023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12249.502913] LNetError: 357023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12249.515562] LNet: Removed LNI 192.168.201.156@tcp [12251.271209] Key type .llcrypt unregistered [12251.274444] Key type ._llcrypt unregistered [12283.025581] Key type ._llcrypt registered [12283.029105] Key type .llcrypt registered [12283.157935] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [12284.690125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12284.725316] alg: No test for adler32 (adler32-zlib) [12285.840809] Lustre: Lustre: Build Version: 2.17.57_1_g3e0b683 [12286.113268] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [12287.828629] Key type lgssc registered [12288.857059] Lustre: Echo OBD driver; http://www.lustre.org/ [12296.971212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12296.982564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.703319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12304.341446] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12308.962878] LustreError: 358591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.981095] LustreError: 358591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12313.497662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12313.880189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12318.415848] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12326.072963] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12331.484265] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12338.788863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12339.304317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12345.644685] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12352.625630] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12364.259927] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12364.286634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12364.292217] Lustre: Skipped 1 previous similar message [12365.530151] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.56@tcp (stopping) [12369.399762] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12369.407821] Lustre: Skipped 1 previous similar message [12369.675794] Lustre: server umount lustre-MDT0001 complete [12370.626268] LustreError: 358588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12374.500330] LustreError: 359320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12374.528331] LustreError: 359320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12379.284394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12379.365925] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12379.373721] Lustre: Found index 1 for lustre-MDT0001, updating log [12379.411451] Lustre: 358581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12379.693763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12384.071404] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12384.755954] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12388.861338] LustreError: 362192:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12388.866513] LustreError: 362192:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12388.872286] LustreError: 358583:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12388.929810] LustreError: 362195:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -17 [12388.939030] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [12388.939030] [12691.941936] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12691.942366] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12691.962205] Lustre: Skipped 1 previous similar message [12691.980037] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12691.985376] Lustre: Skipped 1 previous similar message [12697.060070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12697.065291] Lustre: Skipped 2 previous similar messages [12704.226491] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12704.716272] Lustre: server umount lustre-MDT0001 complete [12707.303752] LustreError: 358591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12707.332943] LustreError: 358591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12712.424251] LustreError: 358587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12712.435059] LustreError: 358587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12717.698865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12717.853490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12717.856188] Lustre: Found index 1 for lustre-MDT0001, updating log [12717.966844] Lustre: 358581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12718.465970] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12723.705195] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12724.432742] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12726.271466] LustreError: 370645:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12726.281427] LustreError: 370645:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12726.296758] LustreError: 358583:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12726.312548] LustreError: 358583:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12726.388243] LustreError: 370651:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [12726.397551] LustreError: 370651:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [12726.409397] LustreError: 370651:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -17 [12726.422804] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.201.156@tcp [12726.422804] [13030.883302] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13030.886677] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13030.901705] Lustre: Skipped 1 previous similar message [13030.907804] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13030.941851] Lustre: Skipped 5 previous similar messages [13037.035520] Lustre: server umount lustre-MDT0001 complete [13041.126122] LustreError: 360142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13041.165931] LustreError: 360142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13049.077225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13049.171637] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13049.175269] Lustre: Found index 1 for lustre-MDT0001, updating log [13049.210423] Lustre: 358581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13049.669339] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13054.972412] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13055.646536] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [13059.068273] LustreError: 379243:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13059.083075] LustreError: 379243:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13059.091094] LustreError: 358583:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13059.093868] LustreError: 358583:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13059.213681] LustreError: 379246:0:(lod_lov.c:210:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13059.233643] LustreError: 379246:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -114 [13059.260304] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13059.260304] [13382.629393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13382.633843] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13382.641330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13382.641339] Lustre: Skipped 4 previous similar messages [13382.699591] Lustre: Skipped 3 previous similar messages [13385.766567] Lustre: server umount lustre-MDT0000 complete [13387.749969] LustreError: 358586:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13387.789049] LustreError: 358586:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [13389.726411] LustreError: 358572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788582922 with bad export cookie 939285489991779371 [13389.750693] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13389.753458] LustreError: 358572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13392.875939] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13404.132326] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13404.439126] Lustre: server umount lustre-MDT0001 complete [13408.698670] Lustre: server umount lustre-OST0000 complete [13418.157387] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [13427.718377] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [13463.936754] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 00:36:36 (1788582996) [13469.707188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13469.930220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13469.954289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13470.036838] Lustre: lustre-MDT0000: new disk, initializing [13470.115428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13470.147074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13474.382822] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [13484.880519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13484.956810] Lustre: 389677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13485.015049] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13485.018386] Lustre: Skipped 1 previous similar message [13485.126792] Lustre: lustre-MDT0001: new disk, initializing [13485.325278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13485.360360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13485.372062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13489.930778] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [13498.749845] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13504.620682] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13511.521049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13511.710286] Lustre: lustre-OST0000: new disk, initializing [13511.713234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13511.721234] Lustre: 391644:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13511.800405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13516.901113] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [13518.471615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13518.489550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13518.646402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13527.472965] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13536.750927] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13536.754179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13536.764057] LustreError: Skipped 1 previous similar message [13536.775154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13536.797687] Lustre: Skipped 6 previous similar messages [13538.073424] Lustre: server umount lustre-MDT0000 complete [13539.311350] LustreError: 390874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13539.349986] LustreError: 390874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [13542.214815] LustreError: 390481:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [13542.225933] LustreError: 389667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788583075 with bad export cookie 939285489991781324 [13542.227344] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13542.240993] LustreError: 389667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13542.507519] Lustre: server umount lustre-MDT0001 complete [13556.730685] Lustre: server umount lustre-OST0000 complete [13585.114199] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [13594.239570] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [13622.585462] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13277 sec ========= 00:39:14 (1788583154) [13624.974864] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:39:16 (1788583156) === [13630.076123] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:39:21 (1788583161) === [13669.273780] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [13671.859703] Key type lgssc unregistered [13672.186454] LNet: 398014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13672.192467] LNetError: 398014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13672.207090] LNet: Removed LNI 192.168.201.156@tcp [13673.130742] Key type .llcrypt unregistered [13673.132485] Key type ._llcrypt unregistered