[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 498660608 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003291] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008570] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011132] LSM: Security Framework initializing [ 0.012049] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.014062] *** VALIDATE selinux *** [ 0.022656] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027398] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] *** VALIDATE tmpfs *** [ 0.031442] *** VALIDATE proc *** [ 0.032246] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.034289] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037028] Spectre V2 : User space: Vulnerable [ 0.038012] Speculative Store Bypass: Vulnerable [ 0.041155] debug: unmapping init [mem 0xffffffff84859000-0xffffffff84860fff] [ 0.044170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045689] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050015] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053299] rcu: Hierarchical SRCU implementation. [ 0.055465] smp: Bringing up secondary CPUs ... [ 0.056563] x86: Booting SMP configuration: [ 0.057030] .... node #0, CPUs: #1 #2 #3 [ 0.060559] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.138544] node 0 deferred pages initialised in 73ms [ 0.142674] devtmpfs: initialized [ 0.144260] x86/mm: Memory block size: 128MB [ 0.147777] gcov: version magic: 0x41383552 [ 0.151366] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153398] pinctrl core: initialized pinctrl subsystem [ 0.155223] [ 0.155804] ************************************************************* [ 0.159015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161011] ** ** [ 0.164013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166009] ** ** [ 0.169012] ** This means that this kernel is built to expose internal ** [ 0.171009] ** IOMMU data structures, which may compromise security on ** [ 0.173013] ** your system. ** [ 0.175011] ** ** [ 0.178013] ** If you see this message and you are not debugging the ** [ 0.181014] ** kernel, report this immediately to your vendor! ** [ 0.183012] ** ** [ 0.186019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.189013] ************************************************************* [ 0.191765] NET: Registered protocol family 16 [ 0.194483] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.204463] cpuidle: using governor menu [ 0.205414] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208529] PCI: Using configuration type 1 for base access [ 0.211129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220059] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223059] cryptd: max_cpu_qlen set to 1000 [ 0.225294] ACPI: Added _OSI(Module Device) [ 0.228040] ACPI: Added _OSI(Processor Device) [ 0.229014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235959] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243554] ACPI: Interpreter enabled [ 0.245070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.247015] ACPI: Using IOAPIC for interrupt routing [ 0.249116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253377] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263539] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.266043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.277247] acpiphp: Slot [2] registered [ 0.279171] acpiphp: Slot [5] registered [ 0.281161] acpiphp: Slot [6] registered [ 0.283109] acpiphp: Slot [7] registered [ 0.285211] acpiphp: Slot [8] registered [ 0.286134] acpiphp: Slot [9] registered [ 0.288104] acpiphp: Slot [10] registered [ 0.289130] acpiphp: Slot [3] registered [ 0.291107] acpiphp: Slot [4] registered [ 0.293113] acpiphp: Slot [11] registered [ 0.295136] acpiphp: Slot [12] registered [ 0.296127] acpiphp: Slot [13] registered [ 0.298091] acpiphp: Slot [14] registered [ 0.301080] acpiphp: Slot [15] registered [ 0.302110] acpiphp: Slot [16] registered [ 0.304096] acpiphp: Slot [17] registered [ 0.305103] acpiphp: Slot [18] registered [ 0.307117] acpiphp: Slot [19] registered [ 0.308106] acpiphp: Slot [20] registered [ 0.310102] acpiphp: Slot [21] registered [ 0.312099] acpiphp: Slot [22] registered [ 0.313105] acpiphp: Slot [23] registered [ 0.315101] acpiphp: Slot [24] registered [ 0.316178] acpiphp: Slot [25] registered [ 0.318121] acpiphp: Slot [26] registered [ 0.319097] acpiphp: Slot [27] registered [ 0.321093] acpiphp: Slot [28] registered [ 0.322093] acpiphp: Slot [29] registered [ 0.324093] acpiphp: Slot [30] registered [ 0.325086] acpiphp: Slot [31] registered [ 0.327075] PCI host bridge to bus 0000:00 [ 0.328022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348955] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352286] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362717] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379588] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382924] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.389820] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.395015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.406999] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.420971] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470356] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.481014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.515016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.527000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.535014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.543063] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.564017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.576715] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.584014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.594014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.608013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.621341] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.628015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.636014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.649023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.658247] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.667015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.674014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.705217] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.708500] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.713448] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.716377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.719216] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.724000] iommu: Default domain type: Passthrough [ 0.725380] SCSI subsystem initialized [ 0.727068] ACPI: bus type USB registered [ 0.729110] usbcore: registered new interface driver usbfs [ 0.732093] usbcore: registered new interface driver hub [ 0.734080] usbcore: registered new device driver usb [ 0.735246] pps_core: LinuxPPS API ver. 1 registered [ 0.738012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.741058] PTP clock support registered [ 0.743061] EDAC MC: Ver: 3.0.0 [ 0.745119] PCI: Using ACPI for IRQ routing [ 0.747872] NetLabel: Initializing [ 0.749016] NetLabel: domain hash size = 128 [ 0.751012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.753109] NetLabel: unlabeled traffic allowed by default [ 0.757075] vgaarb: loaded [ 0.759036] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.761015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.771950] clocksource: Switched to clocksource kvm-clock [ 0.881947] VFS: Disk quotas dquot_6.6.0 [ 0.883622] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.886396] *** VALIDATE ramfs *** [ 0.887760] *** VALIDATE hugetlbfs *** [ 0.889509] pnp: PnP ACPI init [ 0.891885] pnp: PnP ACPI: found 6 devices [ 0.914458] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.918055] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.920595] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.923218] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.925767] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.928654] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.931898] NET: Registered protocol family 2 [ 0.934572] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.939561] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.943548] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.948838] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.952138] TCP: Hash tables configured (established 65536 bind 65536) [ 0.954745] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.957746] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960815] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.964156] NET: Registered protocol family 1 [ 0.966893] RPC: Registered named UNIX socket transport module. [ 0.969463] RPC: Registered udp transport module. [ 0.971429] RPC: Registered tcp transport module. [ 0.973313] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.976117] NET: Registered protocol family 44 [ 0.977574] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.979797] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.982275] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.984388] PCI: CLS 0 bytes, default 64 [ 0.986494] Unpacking initramfs... [ 2.381570] debug: unmapping init [mem 0xffff91e1bcc54000-0xffff91e1bffbffff] [ 2.385888] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.389647] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.392873] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.895155] Initialise system trusted keyrings [ 2.897032] Key type blacklist registered [ 2.899808] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.909868] zbud: loaded [ 2.913358] *** VALIDATE nfs *** [ 2.914382] *** VALIDATE nfs4 *** [ 2.915888] pstore: using deflate compression [ 2.918952] Platform Keyring initialized [ 3.024237] NET: Registered protocol family 38 [ 3.025643] Key type asymmetric registered [ 3.026745] Asymmetric key parser 'x509' registered [ 3.028348] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.030634] io scheduler mq-deadline registered [ 3.032331] io scheduler kyber registered [ 3.033826] io scheduler bfq registered [ 3.035981] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.039293] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.042551] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.045647] ACPI: Power Button [PWRF] [ 3.052331] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.059964] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.076834] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.084569] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.099966] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.130893] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.161903] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.168126] Non-volatile memory driver v1.3 [ 3.170142] Linux agpgart interface v0.103 [ 3.199672] virtio_blk virtio1: [vda] 149840 512-byte logical blocks (76.7 MB/73.2 MiB) [ 3.202268] vda: detected capacity change from 0 to 76718080 [ 3.215761] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.218226] vdb: detected capacity change from 0 to 1073741824 [ 3.237267] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.241407] vdc: detected capacity change from 0 to 2621440000 [ 3.257253] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.260885] vdd: detected capacity change from 0 to 2621440000 [ 3.276339] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.279231] vde: detected capacity change from 0 to 4294967296 [ 3.293657] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.296375] vdf: detected capacity change from 0 to 4294967296 [ 3.302213] libphy: Fixed MDIO Bus: probed [ 3.310392] usbcore: registered new interface driver usbserial_generic [ 3.314443] usbserial: USB Serial support registered for generic [ 3.317523] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.322931] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.325202] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.327988] mousedev: PS/2 mouse device common for all mice [ 3.330510] rtc_cmos 00:05: RTC can wake from S4 [ 3.333166] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.336971] rtc_cmos 00:05: registered as rtc0 [ 3.340116] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.340811] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.343625] intel_pstate: CPU model not supported [ 3.349628] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.353557] hid: raw HID events driver (C) Jiri Kosina [ 3.356347] usbcore: registered new interface driver usbhid [ 3.357961] usbhid: USB HID core driver [ 3.359233] drop_monitor: Initializing network drop monitor service [ 3.361173] Initializing XFRM netlink socket [ 3.362968] NET: Registered protocol family 10 [ 3.366249] Segment Routing with IPv6 [ 3.367450] NET: Registered protocol family 17 [ 3.369253] mpls_gso: MPLS GSO support [ 3.376506] RAS: Correctable Errors collector initialized. [ 3.378838] AVX version of gcm_enc/dec engaged. [ 3.380668] AES CTR mode by8 optimization enabled [ 3.454370] sched_clock: Marking stable (3454333299, 0)->(4399447738, -945114439) [ 3.457161] registered taskstats version 1 [ 3.459410] Loading compiled-in X.509 certificates [ 3.461692] zswap: loaded using pool lzo/zbud [ 3.485990] Key type big_key registered [ 3.499991] Key type encrypted registered [ 3.502314] ima: No TPM chip found, activating TPM-bypass! [ 3.503795] ima: Allocated hash algorithm: sha1 [ 3.505491] ima: No architecture policies found [ 3.507891] evm: Initialising EVM extended attributes: [ 3.510379] evm: security.selinux [ 3.511928] evm: security.ima [ 3.513129] evm: security.capability [ 3.514641] evm: HMAC attrs: 0x1 [ 3.517261] rtc_cmos 00:05: setting system clock to 2026-10-02 17:49:05 UTC (1790963345) [ 3.524412] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 3.527687] debug: unmapping init [mem 0xffffffff84582000-0xffffffff84858fff] [ 3.538097] Write protecting the kernel read-only data: 28672k [ 3.542140] debug: unmapping init [mem 0xffffffff82c03000-0xffffffff82dfffff] [ 3.544870] debug: unmapping init [mem 0xffffffff83514000-0xffffffff835fffff] [ 3.582131] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.592081] systemd[1]: Detected virtualization kvm. [ 3.594916] systemd[1]: Detected architecture x86-64. [ 3.597186] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.623362] systemd[1]: No hostname configured. [ 3.625466] systemd[1]: Set hostname to . [ 3.627925] random: systemd: uninitialized urandom read (16 bytes read) [ 3.630741] systemd[1]: Initializing machine ID from random generator. [ 3.693515] random: ln: uninitialized urandom read (6 bytes read) [ 3.793984] random: systemd: uninitialized urandom read (16 bytes read) [ 3.797287] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.802270] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.807119] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.453496] device-mapper: uevent: version 1.0.3 [ 4.455154] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.242394] random: fast init done [ 5.281812] virtio_net virtio0 ens2: renamed from eth0 [ 5.363240] scsi host0: ata_piix [ 5.374099] scsi host1: ata_piix [ 5.376217] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.379033] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.103858] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.019856] random: crng init done [ 10.021319] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.521084] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.683435] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.945430] SELinux: Disabled at runtime. [ 12.006653] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.017225] systemd[1]: Detected virtualization kvm. [ 12.018823] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.540563] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.543489] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.548534] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.551977] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.554343] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.561645] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.571346] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK [0[ 12.637706] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.058727] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.371282] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.454679] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.567422] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.580920] EDAC sbridge: Ver: 1.1.2 [ 15.041586] Key type dns_resolver registered [ 15.343360] NFS: Registering the id_resolver key type [ 15.345549] Key type id_resolver registered [ 15.347718] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 41.292468] libcfs: loading out-of-tree module taints kernel. [ 41.316941] Key type ._llcrypt registered [ 41.318218] Key type .llcrypt registered [ 41.373853] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 55.771076] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 57.787721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 57.799743] alg: No test for adler32 (adler32-zlib) [ 59.401917] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 60.631364] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 62.472199] Key type lgssc registered [ 65.099981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.531970] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 99.260316] hrtimer: interrupt took 4190165 ns [ 132.316262] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 145.672812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.696450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.019380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.060398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.123363] Lustre: lustre-MDT0000: new disk, initializing [ 147.178649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.191293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.009296] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.979646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.089164] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.118438] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.122760] Lustre: Skipped 1 previous similar message [ 167.210261] Lustre: lustre-MDT0001: new disk, initializing [ 167.284673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.309137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 167.322479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 172.327524] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.047546] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.023963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.216023] Lustre: lustre-OST0000: new disk, initializing [ 185.222441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.227715] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 185.281125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.529938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 188.544222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 188.595346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.273956] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.880812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.016418] Lustre: lustre-OST0001: new disk, initializing [ 204.019502] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.023719] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.079580] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.757306] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.477635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 211.486955] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.544349] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 218.127495] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.653630] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 226.489866] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 230.015974] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 233.154628] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 234.607750] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 236.218904] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 237.282607] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 13:52:58 EDT 2026 [ 246.198517] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 246.936271] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 262.625939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.626754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 262.635554] Lustre: Skipped 1 previous similar message [ 267.745745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 267.749566] Lustre: Skipped 6 previous similar messages [ 272.864849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 272.868855] Lustre: Skipped 3 previous similar messages [ 274.400122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 274.484995] Lustre: server umount lustre-MDT0000 complete [ 276.186458] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963618 with bad export cookie 1978303524996503015 [ 276.189864] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 276.193351] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 277.984698] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 277.985479] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 277.990327] Lustre: Skipped 3 previous similar messages [ 282.017838] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 282.022876] Lustre: Skipped 2 previous similar messages [ 282.377265] Lustre: server umount lustre-MDT0001 complete [ 284.130989] Lustre: server umount lustre-OST0000 complete [ 286.053798] Lustre: server umount lustre-OST0001 complete [ 289.587737] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 297.566889] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 328.808096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 329.051741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 329.083852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 329.151860] Lustre: lustre-MDT0000: new disk, initializing [ 329.207628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 329.224702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 333.588316] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 341.728658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 342.198363] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 342.260570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 342.263656] Lustre: Skipped 1 previous similar message [ 342.338181] Lustre: lustre-MDT0001: new disk, initializing [ 342.459092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 342.508022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 342.524527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 346.721972] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 357.221708] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 364.185174] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 371.059406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 371.294226] Lustre: lustre-OST0000: new disk, initializing [ 371.304505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 371.314504] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 371.404617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.059987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 373.081437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 373.288877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 376.680347] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 385.079719] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 389.798613] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 389.925525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 393.763882] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 393.984224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 398.820457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 398.826550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.863092] Lustre: Skipped 1 previous similar message [ 398.865706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 401.621103] Lustre: server umount lustre-OST0000 complete [ 409.059960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 409.064691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 409.076914] Lustre: Skipped 1 previous similar message [ 410.720449] Lustre: server umount lustre-MDT0000 complete [ 413.013426] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963754 with bad export cookie 1978303524996504548 [ 413.019163] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 413.021356] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 413.293148] Lustre: server umount lustre-MDT0001 complete [ 416.166899] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 13:55:57 (1790963757) [ 420.090535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.371926] LustreError: 19311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 420.387871] LustreError: 19311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 420.427656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 423.200947] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 425.442807] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.245167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.922225] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 435.642234] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 440.239976] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 449.073960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.051901] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 471.203827] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 475.492631] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 484.835945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 484.853234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.890615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.906726] Lustre: Skipped 3 previous similar messages [ 490.296395] Lustre: server umount lustre-MDT0000 complete [ 490.977955] LustreError: 20038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.992613] LustreError: 20038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 493.883694] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963835 with bad export cookie 1978303524996505423 [ 493.886304] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.895057] LustreError: 19293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 494.438774] Lustre: server umount lustre-MDT0001 complete [ 507.428750] Lustre: server umount lustre-OST0000 complete [ 526.726758] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 535.100554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.432374] LustreError: 23692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.511879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.526455] Lustre: Skipped 2 previous similar messages [ 540.246493] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 540.641885] LustreError: 23693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.630775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 553.354884] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 555.910693] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 561.737461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 567.779866] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 576.080733] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 576.227894] Lustre: lustre-OST0001: new disk, initializing [ 576.230699] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 576.238823] Lustre: 26004:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 578.076664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 578.094261] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 578.209234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 582.733491] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 591.591289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 595.127652] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 610.786360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 610.794483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.800467] Lustre: Skipped 2 previous similar messages [ 610.804740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 610.810366] Lustre: Skipped 3 previous similar messages [ 612.453692] Lustre: server umount lustre-MDT0000 complete [ 615.350763] LustreError: 23673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790963957 with bad export cookie 1978303524996506270 [ 615.353818] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.356712] LustreError: 23673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 615.654023] Lustre: server umount lustre-MDT0001 complete [ 629.048599] Lustre: server umount lustre-OST0000 complete [ 632.295367] Lustre: 3648:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790963958/real 1790963958] req@ffff91e104c97800 x1877961240443392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790963974 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 642.726274] Lustre: server umount lustre-OST0001 complete [ 656.860957] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 666.651368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.109749] LustreError: 29026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.137762] LustreError: 29026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 667.224378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.232026] Lustre: Skipped 3 previous similar messages [ 672.343932] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 680.444498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.406372] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 690.081687] Lustre: 30176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 698.473904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 705.456355] LustreError: 30533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 705.482217] LustreError: 30533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 706.786077] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 715.018981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 722.226573] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 729.288382] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 732.643161] Lustre: 32068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 741.352097] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.367361] Lustre: Skipped 5 previous similar messages [ 741.374906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 741.390752] Lustre: Skipped 7 previous similar messages [ 745.348880] Lustre: server umount lustre-MDT0000 complete [ 746.475043] LustreError: 29027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 746.490895] LustreError: 29027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 748.749493] LustreError: 29006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964090 with bad export cookie 1978303524996507565 [ 748.757264] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 748.770959] LustreError: 29006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 749.067758] Lustre: server umount lustre-MDT0001 complete [ 763.005402] Lustre: server umount lustre-OST0000 complete [ 777.221160] Lustre: server umount lustre-OST0001 complete [ 785.849783] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 14:02:06 (1790964126) [ 811.868105] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 820.469788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.758649] LustreError: 34871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 824.006048] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 830.787155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.000381] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 837.213247] Lustre: 36016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 842.372220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 847.472507] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 853.715894] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 858.548994] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 863.940603] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 866.299664] Lustre: 37896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 871.395458] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.401655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 871.413160] Lustre: Skipped 2 previous similar messages [ 871.423185] Lustre: Skipped 1 previous similar message [ 875.844001] Lustre: server umount lustre-MDT0000 complete [ 878.942734] LustreError: 34853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964220 with bad export cookie 1978303524996508678 [ 878.946987] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 878.956741] LustreError: 34853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 879.384121] Lustre: server umount lustre-MDT0001 complete [ 892.230618] Lustre: server umount lustre-OST0000 complete [ 905.848766] Lustre: server umount lustre-OST0001 complete [ 915.556147] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 14:04:16 (1790964256) [ 941.441755] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 950.736263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.093511] LustreError: 40065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.115809] LustreError: 40065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 951.171461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.176650] Lustre: Skipped 7 previous similar messages [ 954.856111] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 962.763522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.044402] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 969.459214] Lustre: 41216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 975.705953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 980.918281] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 987.761265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 993.756071] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1000.656309] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1003.696673] Lustre: 43093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1004.377962] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1004.382521] Lustre: Skipped 1 previous similar message [ 1009.123053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1009.135178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.151756] Lustre: Skipped 1 previous similar message [ 1013.354378] Lustre: server umount lustre-MDT0000 complete [ 1016.412828] LustreError: 40045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964358 with bad export cookie 1978303524996509784 [ 1016.416995] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.425148] LustreError: 40045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1022.732991] Lustre: server umount lustre-MDT0001 complete [ 1029.830951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.400919] LustreError: 3647:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff91e10199f100 x1877961240590080/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1046.182581] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1052.915405] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1054.678859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.209625] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1060.344814] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1060.351680] Lustre: Skipped 1 previous similar message [ 1067.840871] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1069.284478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1074.677581] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1076.046361] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1080.812458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1085.953293] Lustre: server umount lustre-MDT0000 complete [ 1089.978365] LustreError: 40045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964431 with bad export cookie 1978303524996510911 [ 1089.989416] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.996525] LustreError: 40045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1090.357193] Lustre: server umount lustre-MDT0001 complete [ 1104.593901] Lustre: server umount lustre-OST0000 complete [ 1107.553126] Lustre: 3650:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790964433/real 1790964433] req@ffff91e10199df80 x1877961240622464/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790964449 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1107.868920] Lustre: server umount lustre-OST0001 complete [ 1114.225632] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 14:07:35 (1790964455) [ 1125.598185] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1134.516623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.643692] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1145.624298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.704977] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1151.922359] Lustre: 48848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1157.627974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1162.971927] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1171.013684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1176.638138] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1184.250378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1188.639364] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1202.147499] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.148915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1202.153901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.153910] Lustre: Skipped 15 previous similar messages [ 1202.186095] Lustre: Skipped 11 previous similar messages [ 1207.276207] LustreError: 47693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.292172] LustreError: 47693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 1207.383649] Lustre: server umount lustre-MDT0000 complete [ 1211.093456] LustreError: 47678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790964553 with bad export cookie 1978303524996511702 [ 1211.107234] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1211.118060] LustreError: 47678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1211.499322] Lustre: server umount lustre-MDT0001 complete [ 1225.755785] Lustre: server umount lustre-OST0000 complete [ 1228.384818] Lustre: 3651:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790964554/real 1790964554] req@ffff91e23f4b8700 x1877961240676352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790964570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1229.522910] Lustre: server umount lustre-OST0001 complete [ 1240.928308] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 14:09:41 (1790964581) [ 1243.276646] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1245.078478] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 14:09:45 (1790964585) [ 1256.338269] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1264.950430] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1285.958303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.274435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1286.411828] Lustre: lustre-MDT0000: new disk, initializing [ 1286.523273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1290.752556] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1299.833922] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1306.333484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1306.668472] Lustre: lustre-OST0000: new disk, initializing [ 1306.674447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1306.691335] Lustre: Skipped 1 previous similar message [ 1306.695815] Lustre: 54933:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1308.553364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1308.564440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1308.604831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1312.048472] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1320.690654] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1322.827553] Lustre: DEBUG MARKER: create test files [ 1364.965094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.210165] Lustre: server umount lustre-OST0000 complete [ 1371.737638] Lustre: server umount lustre-MDT0000 complete [ 1384.226178] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1388.367799] Key type lgssc unregistered [ 1388.674523] LNet: 56563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.686991] LNetError: 56563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.712748] LNet: Removed LNI 192.168.203.155@tcp [ 1390.131172] Key type .llcrypt unregistered [ 1390.134310] Key type ._llcrypt unregistered [ 1420.563794] Key type ._llcrypt registered [ 1420.567823] Key type .llcrypt registered [ 1420.721278] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1421.453838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1421.576263] alg: No test for adler32 (adler32-zlib) [ 1422.620133] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1422.924969] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1424.688166] Key type lgssc registered [ 1425.693719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1432.454495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1432.516921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.896315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.950480] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1444.876533] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1450.307856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1450.787216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.675527] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1459.020872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:100 to 0x240000400:129) [ 1459.251409] Lustre: 59289:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 510 < left 544, rollback = 9 [ 1459.273026] Lustre: 59289:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/5, destroy: 0/0/0 [ 1459.286297] Lustre: 59289:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1459.298763] Lustre: 59289:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1459.310469] Lustre: 59289:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1459.315923] Lustre: 59289:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1467.024418] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1472.956534] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1479.579064] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1488.493609] Lustre: server umount lustre-OST0000 complete [ 1501.306643] Lustre: server umount lustre-MDT0000 complete [ 1511.532318] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1514.585701] Key type lgssc unregistered [ 1514.888754] LNet: 60898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.895585] LNetError: 60898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.909631] LNet: Removed LNI 192.168.203.155@tcp [ 1515.912372] Key type .llcrypt unregistered [ 1515.915335] Key type ._llcrypt unregistered [ 1543.624814] Key type ._llcrypt registered [ 1543.628777] Key type .llcrypt registered [ 1543.688125] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1544.492210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1544.667376] alg: No test for adler32 (adler32-zlib) [ 1545.819978] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1546.213353] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1547.904159] Key type lgssc registered [ 1549.325671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1557.797581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1557.812347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.219945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1563.980250] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1571.800409] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1578.390706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1578.738808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.565775] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1584.889683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1584.952522] Lustre: 63730:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1584.962584] Lustre: 63730:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1584.973904] Lustre: 63730:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1584.982667] Lustre: 63730:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1584.987557] Lustre: 63730:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1584.996475] Lustre: 63730:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1591.105705] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1596.439162] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1605.604564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1605.623575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.648268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1610.726648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1611.221482] Lustre: server umount lustre-OST0000 complete [ 1615.670816] Lustre: server umount lustre-MDT0000 complete [ 1626.002616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1629.043335] Key type lgssc unregistered [ 1629.411427] LNet: 65289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.418660] LNetError: 65289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.453262] LNet: Removed LNI 192.168.203.155@tcp [ 1630.225822] Key type .llcrypt unregistered [ 1630.228381] Key type ._llcrypt unregistered [ 1647.696731] Key type ._llcrypt registered [ 1647.699800] Key type .llcrypt registered [ 1647.798854] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1662.436613] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1663.517126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1663.531554] alg: No test for adler32 (adler32-zlib) [ 1664.566730] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 1664.770236] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1666.448321] Key type lgssc registered [ 1667.576657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1700.636642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1700.695112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1701.900284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1701.929491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1702.005867] Lustre: lustre-MDT0000: new disk, initializing [ 1702.082237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.095395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1706.665873] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1717.392197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1717.520704] Lustre: 68438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1717.583345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1717.587465] Lustre: Skipped 1 previous similar message [ 1717.716674] Lustre: lustre-MDT0001: new disk, initializing [ 1717.877560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1717.921036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1717.936891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1723.007618] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1731.597359] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1736.848752] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1742.953457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1743.263629] Lustre: lustre-OST0000: new disk, initializing [ 1743.268407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1743.283262] Lustre: 70407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1743.383233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.761964] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1751.652678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1751.686772] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1751.782782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1759.348092] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1763.755760] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1764.039630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1767.942899] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1768.128660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1772.517640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1772.518130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1772.530365] LustreError: Skipped 1 previous similar message [ 1772.533429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1772.542514] Lustre: Skipped 1 previous similar message [ 1772.574821] Lustre: Skipped 1 previous similar message [ 1777.636346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1777.655706] Lustre: Skipped 1 previous similar message [ 1782.768667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.783617] Lustre: Skipped 1 previous similar message [ 1784.288350] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1784.430986] Lustre: server umount lustre-OST0000 complete [ 1787.873888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.889982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1792.994198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1793.001759] Lustre: Skipped 1 previous similar message [ 1794.091401] Lustre: server umount lustre-MDT0000 complete [ 1797.262638] LustreError: 69235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965139 with bad export cookie 2591528483052112802 [ 1797.267642] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1797.272941] LustreError: 69235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1797.503615] Lustre: server umount lustre-MDT0001 complete [ 1807.230832] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 14:19:08 (1790965148) [ 1818.000691] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1858.274987] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 1867.828731] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1901.868514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.130339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1902.192422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1902.290421] Lustre: lustre-MDT0000: new disk, initializing [ 1902.363827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.378109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1907.139785] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1917.492857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.629515] Lustre: 76426:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1917.708730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1917.718913] Lustre: Skipped 1 previous similar message [ 1917.854722] Lustre: lustre-MDT0001: new disk, initializing [ 1918.029517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.091554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1918.102306] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1922.492448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1931.317801] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1936.973363] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1944.898942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1945.314648] Lustre: lustre-OST0000: new disk, initializing [ 1945.320961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1945.328895] Lustre: 78396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1945.444646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1952.629365] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1953.398611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1953.430197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1953.589700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1962.347793] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1967.658915] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1967.955882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1972.045920] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1972.278269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1974.260341] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1974.263128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.269462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1974.293693] Lustre: Skipped 2 previous similar messages [ 1988.064285] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1988.205259] Lustre: server umount lustre-OST0000 complete [ 1994.721845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.725989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1994.737989] Lustre: Skipped 1 previous similar message [ 1994.742285] Lustre: Skipped 5 previous similar messages [ 1998.478320] Lustre: server umount lustre-MDT0000 complete [ 1999.842418] LustreError: 77376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1999.861279] LustreError: 77376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2001.850483] LustreError: 76417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965343 with bad export cookie 2591528483052113684 [ 2001.853205] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2001.859360] LustreError: 76417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2002.215084] Lustre: server umount lustre-MDT0001 complete [ 2010.739438] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 14:22:31 (1790965351) [ 2017.333687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.865639] LustreError: 80810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.937160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.792072] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2023.396139] LustreError: 80811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2027.503189] LustreError: 80810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.823848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.668575] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2044.550862] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.857832] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2057.824680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2058.167280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.176990] Lustre: Skipped 1 previous similar message [ 2064.295676] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2070.445191] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2073.636083] Lustre: 83480:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 60ae0f9b-9175-4bdd-8411-6c10e808f712 at adminstrative request [ 2076.630764] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2076.652125] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2086.898409] Lustre: MGS: Client 60ae0f9b-9175-4bdd-8411-6c10e808f712 (at 0@lo) reconnecting [ 2086.906126] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x23f6f56cc1a0d286 to 0x23f6f56cc1a0d526 [ 2086.921329] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 2111.244045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2111.485993] Lustre: lustre-OST0001: new disk, initializing [ 2111.488852] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2111.497521] Lustre: 83874:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2117.792058] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2119.323817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2119.343420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2119.478341] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2128.018845] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2135.010099] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2135.023327] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.035891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2135.040805] Lustre: Skipped 2 previous similar messages [ 2145.248131] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2145.413038] Lustre: server umount lustre-OST0001 complete [ 2150.375356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2150.386148] LustreError: 82631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.392533] Lustre: Skipped 2 previous similar messages [ 2150.409652] LustreError: 82631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2155.856534] Lustre: server umount lustre-OST0000 complete [ 2165.960748] Lustre: server umount lustre-MDT0000 complete [ 2170.356029] LustreError: 83873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965512 with bad export cookie 2591528483052115238 [ 2170.357105] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2170.361768] LustreError: 83873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2170.686747] Lustre: server umount lustre-MDT0001 complete [ 2180.450616] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 14:25:20 (1790965520) [ 2182.279506] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2185.362607] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 14:25:25 (1790965525) [ 2187.260533] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2190.661547] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 14:25:30 (1790965530) [ 2193.191189] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2196.129330] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 14:25:36 (1790965536) [ 2199.050654] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2202.652347] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 14:25:42 (1790965542) [ 2205.438146] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2213.145228] loop: module loaded [ 2213.164801] loop0: detected capacity change from 0 to 4294967296 [ 2213.219492] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2214.887051] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2219.025810] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2222.821906] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2249.959035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2250.008759] LustreError: 87999:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2250.019572] LustreError: 87999:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2250.027220] LustreError: 87999:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2250.032685] LustreError: 87999:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2268.794288] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2281.484232] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2318.894073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.128289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2319.146157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2319.265419] Lustre: lustre-MDT0000: new disk, initializing [ 2319.359023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.363463] Lustre: Skipped 1 previous similar message [ 2319.382077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2324.289432] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2334.418595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.541263] Lustre: 90867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2334.822079] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2339.113340] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2347.111025] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2352.612301] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2358.233349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2358.462842] Lustre: 92838:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2364.014239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2364.059311] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2364.149360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2373.426303] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2377.486687] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2377.685235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2381.250555] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2381.432907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2384.866958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2384.868356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.870840] LustreError: Skipped 2 previous similar messages [ 2384.871924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2384.871930] Lustre: Skipped 12 previous similar messages [ 2384.937971] Lustre: Skipped 3 previous similar messages [ 2389.432234] Lustre: server umount lustre-OST0000 complete [ 2400.432696] Lustre: server umount lustre-MDT0000 complete [ 2404.322834] LustreError: 90876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.370563] LustreError: 90876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2404.599410] LustreError: 92837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790965746 with bad export cookie 2591528483052115854 [ 2404.601736] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2404.635681] LustreError: 92837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2405.153138] Lustre: server umount lustre-MDT0001 complete [ 2414.411792] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 14:29:14 (1790965754) [ 2421.621925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2422.328815] LustreError: 95249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.028510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2435.317776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.990638] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2446.856438] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2453.100799] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2460.518292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2461.089213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2461.102897] Lustre: Skipped 4 previous similar messages [ 2468.927975] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2476.050943] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2481.223954] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2481.508542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2485.835281] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2486.092307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2491.869535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2492.052891] Lustre: lustre-OST0001: new disk, initializing [ 2492.062740] Lustre: Skipped 2 previous similar messages [ 2492.066978] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2492.074163] Lustre: Skipped 3 previous similar messages [ 2492.080768] Lustre: 98623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2497.636975] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2502.264555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2502.274067] Lustre: Skipped 2 previous similar messages [ 2502.282705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2502.439960] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2505.346583] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2510.188794] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2510.410942] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2515.429633] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2515.669061] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2534.556854] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2556.494377] Lustre: Failing over lustre-MDT0000 [ 2558.433410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2558.444915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.467297] Lustre: Skipped 2 previous similar messages [ 2558.473403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.485163] Lustre: Skipped 5 previous similar messages [ 2558.954276] LustreError: 99876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.960636] Lustre: server umount lustre-MDT0000 complete [ 2558.969559] LustreError: 99876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2574.310992] Lustre: 66663:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790965900/real 1790965900] req@00000000df8ed38a x1877962922400896/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790965916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2574.340905] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.471398] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2582.475660] LDISKFS-fs (dm-0): recovery complete [ 2582.482217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.547052] LustreError: 66660:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000070e867d5 x1877962922407680/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2584.940810] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2586.490598] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2586.492931] Lustre: 99876:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2589.389208] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2590.189590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2590.196566] Lustre: 95249:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2590.207844] Lustre: 95249:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 2590.297290] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2590.624088] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2590.988537] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2591.001415] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2591.321751] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2591.345180] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 1 previous similar message [ 2591.364907] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2591.378304] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 7 previous similar messages [ 2592.025537] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2592.039929] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2592.360110] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2592.368913] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 2 previous similar messages [ 2593.653327] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2593.671603] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 6 previous similar messages [ 2594.307778] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2594.319819] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2594.640125] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2594.647766] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 6 previous similar messages [ 2597.843029] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2597.856144] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 12 previous similar messages [ 2598.488948] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2598.500805] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2598.824190] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2598.830592] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 12 previous similar messages [ 2606.145032] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2606.164147] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 25 previous similar messages [ 2606.501167] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2606.510758] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2606.833861] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2606.842375] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 24 previous similar messages [ 2622.316209] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2622.328921] Lustre: 101481:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 50 previous similar messages [ 2622.649359] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2622.659306] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2622.984134] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2622.994310] LustreError: 101481:0:(ldlm_lib.c:2723:replay_request_or_update()) Skipped 50 previous similar messages [ 2646.128315] Lustre: 101481:0:(ldlm_lib.c:2419:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 2646.135559] Lustre: 101481:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2646.146031] Lustre: 101481:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client dd9dc90a-2dae-40d9-b284-1e8c83823462@192.168.203.55@tcp [ 2646.158119] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 2646.163964] Lustre: 101481:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@00000000128c9281 x1877962907519616/t0(8589934772) o35->dd9dc90a-2dae-40d9-b284-1e8c83823462@192.168.203.55@tcp:298/0 lens 392/0 e 0 to 0 dl 1790965998 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 2646.202587] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2646.215890] Lustre: lustre-MDT0000: Denying connection for new client dd9dc90a-2dae-40d9-b284-1e8c83823462 (at 192.168.203.55@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 2646.248244] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2646.284200] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2646.291190] Lustre: Skipped 2 previous similar messages [ 2646.299423] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 2646.345376] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:46 to 0x2c0000401:545) [ 2646.356533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 2659.809677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2659.818968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.848390] Lustre: Skipped 3 previous similar messages [ 2665.335511] Lustre: server umount lustre-OST0000 complete [ 2666.985844] LustreError: 97072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.008564] LustreError: 97072:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 33 previous similar messages [ 2683.360556] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2683.516846] Lustre: server umount lustre-OST0001 complete [ 2687.457414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2687.466595] LustreError: Skipped 1 previous similar message [ 2701.280462] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2701.612884] Lustre: server umount lustre-MDT0000 complete [ 2705.116549] LustreError: 95230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966047 with bad export cookie 2591528483052167066 [ 2705.126658] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.136758] LustreError: 95230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2705.424279] Lustre: server umount lustre-MDT0001 complete [ 2713.352991] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 14:34:14 (1790966054) [ 2720.998649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2721.019790] Lustre: *** cfs_fail_loc=197, val=0*** [ 2736.737443] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2741.856431] LustreError: 103458:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2746.996455] Lustre: *** cfs_fail_loc=197, val=0*** [ 2746.999687] Lustre: Skipped 65 previous similar messages [ 2767.587754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.595870] Lustre: Skipped 2 previous similar messages [ 2773.841059] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2780.558659] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2782.839924] Lustre: server umount lustre-OST0000 complete [ 2791.687928] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 14:35:32 (1790966132) [ 2811.861529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.291274] LustreError: 105076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.305500] LustreError: 105076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2815.659626] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2824.957539] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2829.274428] LustreError: 105107:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2829.300670] LustreError: 105107:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2829.319370] LustreError: 105107:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2829.869812] Lustre: server umount lustre-MDT0000 complete [ 2859.668685] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2868.920678] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2904.022948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2904.284664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2904.322965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2904.414707] Lustre: lustre-MDT0000: new disk, initializing [ 2904.530790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2909.044359] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2920.498623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.603743] Lustre: 109285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2920.873290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2924.788982] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2935.109388] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2940.906637] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2946.378081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2946.637727] Lustre: 111251:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2952.447673] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2955.964650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2961.631022] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2965.980591] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2966.216409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2971.084358] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2971.318177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2976.229249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2976.238612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.250785] Lustre: Skipped 5 previous similar messages [ 2976.256613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2976.264527] Lustre: Skipped 16 previous similar messages [ 2979.660767] Lustre: server umount lustre-OST0000 complete [ 2989.649117] Lustre: server umount lustre-MDT0000 complete [ 2994.291513] LustreError: 109278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966336 with bad export cookie 2591528483052177685 [ 2994.293247] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2994.309053] LustreError: 109278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2994.866878] Lustre: server umount lustre-MDT0001 complete [ 3004.710932] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 14:39:05 (1790966345) [ 3014.692536] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3017.687720] Key type lgssc unregistered [ 3018.008938] LNet: 113740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3018.021397] LNetError: 113740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3019.048311] LNet: Removed LNI 192.168.203.155@tcp [ 3019.962153] Key type .llcrypt unregistered [ 3019.964911] Key type ._llcrypt unregistered [ 3035.991746] Key type ._llcrypt registered [ 3035.995528] Key type .llcrypt registered [ 3036.083717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3051.316867] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3052.405894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3052.469164] alg: No test for adler32 (adler32-zlib) [ 3053.514443] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3053.765423] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3055.478684] Key type lgssc registered [ 3056.726675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3094.357842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3094.379852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3095.676924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3095.736327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3095.879545] Lustre: lustre-MDT0000: new disk, initializing [ 3095.979993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3096.014041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3100.579756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3110.373206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.635760] Lustre: lustre-OST0000: new disk, initializing [ 3110.640039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3110.646184] Lustre: Skipped 1 previous similar message [ 3110.651630] Lustre: 117908:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3110.737622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.974954] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3119.216279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3119.227892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3119.253137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3128.359151] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3128.601323] Lustre: lustre-OST0001: new disk, initializing [ 3128.607977] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3128.626704] Lustre: 118772:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3128.734915] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3134.641898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3134.658376] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3134.709649] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3135.511559] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3145.191045] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.216398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.232048] Lustre: Skipped 1 previous similar message [ 3150.114890] Lustre: server umount lustre-MDT0000 complete [ 3156.589245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3161.157584] LustreError: 117096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966503 with bad export cookie 16244694556650792358 [ 3161.163491] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3161.182409] LustreError: 117096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3161.333784] Lustre: server umount lustre-OST0000 complete [ 3165.568481] Lustre: server umount lustre-OST0001 complete [ 3177.121376] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 14:41:57 (1790966517) [ 3204.126729] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 14:42:24 (1790966544) [ 3213.870086] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3236.085589] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 14:42:57 (1790966577) [ 3270.574539] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3281.346430] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3314.292439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3314.482938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3314.501273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3314.567970] Lustre: lustre-MDT0000: new disk, initializing [ 3314.635377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.651441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3318.901850] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3328.596943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.742288] Lustre: 126527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3328.819544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3328.823504] Lustre: Skipped 1 previous similar message [ 3328.898827] Lustre: lustre-MDT0001: new disk, initializing [ 3329.065696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3329.126970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3329.148412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3334.973611] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3345.237961] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3351.308557] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3358.265053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3358.555630] Lustre: lustre-OST0000: new disk, initializing [ 3358.561456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3358.569886] Lustre: 128495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3358.663591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.988649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3364.999866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3365.147139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3365.421206] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3376.393280] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3381.783291] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3382.034794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3386.777788] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3387.065653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3390.949121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3390.957585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.965392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3390.975538] Lustre: Skipped 2 previous similar messages [ 3396.087597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.105794] Lustre: Skipped 2 previous similar messages [ 3401.188586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.191354] Lustre: Skipped 1 previous similar message [ 3403.234899] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3403.386830] Lustre: server umount lustre-OST0000 complete [ 3410.918771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3410.923199] LustreError: Skipped 1 previous similar message [ 3410.942844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3410.951188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3413.671907] Lustre: server umount lustre-MDT0000 complete [ 3419.015474] LustreError: 126518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966760 with bad export cookie 16244694556650793583 [ 3419.024211] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.032229] LustreError: 126518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3419.283723] Lustre: server umount lustre-MDT0001 complete [ 3428.591062] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 14:46:09 (1790966769) [ 3436.315472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.730626] LustreError: 130909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.766482] LustreError: 130909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3436.850117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.243953] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3442.145085] LustreError: 130908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.243889] LustreError: 130909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.570907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.794491] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3463.737824] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3469.970507] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3476.473469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.816632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.826404] Lustre: Skipped 1 previous similar message [ 3482.432519] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3488.806704] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3496.734395] Lustre: *** cfs_fail_loc=159, val=0*** [ 3513.256119] Lustre: lustre-MDT0000: Client 6e5f0734-8adc-4bea-a3e8-c5794f63d6c3 (at 192.168.203.55@tcp) reconnecting [ 3515.103842] Lustre: *** cfs_fail_loc=159, val=0*** [ 3515.106961] Lustre: Skipped 5 previous similar messages [ 3530.652263] Lustre: lustre-MDT0000: Client 6e5f0734-8adc-4bea-a3e8-c5794f63d6c3 (at 192.168.203.55@tcp) reconnecting [ 3548.641577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3548.644814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.666937] Lustre: Skipped 1 previous similar message [ 3548.674486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3548.677242] Lustre: Skipped 2 previous similar messages [ 3550.689222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3554.306682] Lustre: server umount lustre-OST0000 complete [ 3560.929387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.951350] Lustre: Skipped 2 previous similar messages [ 3564.273210] Lustre: server umount lustre-MDT0000 complete [ 3566.050179] LustreError: 130908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.077857] LustreError: 130908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3567.719234] LustreError: 130887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790966909 with bad export cookie 16244694556650794465 [ 3567.734362] LustreError: 130887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3567.735587] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3568.052953] Lustre: server umount lustre-MDT0001 complete [ 3578.137764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3580.848237] Key type lgssc unregistered [ 3581.138880] LNet: 134687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3581.153702] LNetError: 134687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3581.176345] LNet: Removed LNI 192.168.203.155@tcp [ 3581.974975] Key type .llcrypt unregistered [ 3581.976530] Key type ._llcrypt unregistered [ 3607.917557] Key type ._llcrypt registered [ 3607.919268] Key type .llcrypt registered [ 3608.096626] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3608.949800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3608.998158] alg: No test for adler32 (adler32-zlib) [ 3610.190499] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3610.353513] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3612.024222] Key type lgssc registered [ 3613.595274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3621.802937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3621.838502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.310620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.993440] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3633.647027] LustreError: 136102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.659197] LustreError: 136102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3636.344487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.794338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3641.175719] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3648.225622] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3654.576546] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3661.516864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3661.905224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3667.433250] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3669.104578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3674.186728] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3692.425674] Lustre: *** cfs_fail_loc=159, val=0*** [ 3707.857813] Lustre: lustre-MDT0000: Client 879c833b-7b31-431a-bd37-cf3c04ec2d80 (at 192.168.203.55@tcp) reconnecting [ 3710.521346] Lustre: *** cfs_fail_loc=159, val=0*** [ 3725.863257] Lustre: lustre-MDT0001: Client 879c833b-7b31-431a-bd37-cf3c04ec2d80 (at 192.168.203.55@tcp) reconnecting [ 3727.606610] Lustre: *** cfs_fail_loc=159, val=0*** [ 3727.607971] Lustre: Skipped 3 previous similar messages [ 3744.174294] Lustre: lustre-MDT0001: Client 879c833b-7b31-431a-bd37-cf3c04ec2d80 (at 192.168.203.55@tcp) reconnecting [ 3749.501839] Lustre: 139160:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3752.069782] Lustre: *** cfs_fail_loc=159, val=0*** [ 3752.076833] Lustre: Skipped 4 previous similar messages [ 3767.816212] Lustre: lustre-MDT0000: Client f2f1e01d-6a32-49b5-b371-aff225946922 (at 192.168.203.55@tcp) reconnecting [ 3769.610235] Lustre: *** cfs_fail_loc=159, val=0*** [ 3769.614376] Lustre: Skipped 14 previous similar messages [ 3786.247464] Lustre: lustre-MDT0000: Client f2f1e01d-6a32-49b5-b371-aff225946922 (at 192.168.203.55@tcp) reconnecting [ 3787.908633] Lustre: 139453:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3807.716316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3807.732332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.756797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3809.763979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3809.768256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3809.769629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3811.648541] Lustre: server umount lustre-OST0000 complete [ 3815.905697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3815.910805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.919719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3815.930467] Lustre: Skipped 1 previous similar message [ 3820.002729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.004936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3820.030867] Lustre: Skipped 1 previous similar message [ 3821.571547] Lustre: server umount lustre-MDT0000 complete [ 3825.121662] LustreError: 139339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3825.140067] LustreError: 139339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3825.534546] LustreError: 136081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967167 with bad export cookie 17474044259163533130 [ 3825.537100] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3825.551862] LustreError: 136081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3825.901663] Lustre: server umount lustre-MDT0001 complete [ 3837.385555] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3841.131204] Key type lgssc unregistered [ 3841.535846] LNet: 140533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3841.546723] LNetError: 140533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3841.574751] LNet: Removed LNI 192.168.203.155@tcp [ 3842.539709] Key type .llcrypt unregistered [ 3842.542153] Key type ._llcrypt unregistered [ 3867.233117] Key type ._llcrypt registered [ 3867.236312] Key type .llcrypt registered [ 3867.397176] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3868.125525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3868.251892] alg: No test for adler32 (adler32-zlib) [ 3869.523645] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 3869.804271] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3871.528335] Key type lgssc registered [ 3872.669660] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3879.659759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3879.693293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.209108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3886.375437] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3891.682250] LustreError: 141944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.702405] LustreError: 141944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3896.722062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.802598] LustreError: 141945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3897.231730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3902.441709] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3909.271961] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3914.359539] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3921.467467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3921.831236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.868887] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 3926.964669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 3927.828280] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3936.174766] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3942.079514] Lustre: 144522:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3945.821723] Lustre: 144571:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3947.873951] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 3948.987493] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 3952.297431] Lustre: 144717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 3953.323573] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 3954.351676] Lustre: 144816:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 3957.730369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3957.736330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.738576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.753014] Lustre: Skipped 1 previous similar message [ 3962.857754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3962.884405] Lustre: Skipped 2 previous similar messages [ 3967.975271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3967.984258] Lustre: Skipped 1 previous similar message [ 3972.068045] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3972.335593] Lustre: server umount lustre-OST0000 complete [ 3978.210607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3978.232535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3978.249481] Lustre: Skipped 1 previous similar message [ 3983.331653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3983.348730] Lustre: Skipped 1 previous similar message [ 3983.671637] Lustre: server umount lustre-MDT0000 complete [ 3987.927154] LustreError: 144620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967329 with bad export cookie 6490759843993458258 [ 3987.934202] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.942690] LustreError: 144620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3988.294780] Lustre: server umount lustre-MDT0001 complete [ 4000.002534] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4002.588833] Key type lgssc unregistered [ 4002.889119] LNet: 145877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4002.900306] LNetError: 145877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4002.912462] LNet: Removed LNI 192.168.203.155@tcp [ 4003.862787] Key type .llcrypt unregistered [ 4003.866423] Key type ._llcrypt unregistered [ 4027.810818] Key type ._llcrypt registered [ 4027.814920] Key type .llcrypt registered [ 4027.954217] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4028.752828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4028.765402] alg: No test for adler32 (adler32-zlib) [ 4029.855101] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 4030.058760] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4031.712184] Key type lgssc registered [ 4032.808676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4040.721786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4040.758318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4042.340049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4047.233298] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4052.449686] LustreError: 147291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.464537] LustreError: 147291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4056.658869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.435519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4064.030356] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4071.705820] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4078.889598] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4084.145640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4084.544030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.599286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4089.941833] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4096.678516] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4102.961900] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4106.389101] Lustre: *** cfs_fail_loc=159, val=0*** [ 4122.049521] Lustre: lustre-MDT0000: Client 73c55aaf-edc2-41be-bd90-0621cc9d8555 (at 192.168.203.55@tcp) reconnecting [ 4144.103379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4144.104455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.117650] LustreError: Skipped 1 previous similar message [ 4144.118834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4144.180407] Lustre: Skipped 1 previous similar message [ 4145.747696] Lustre: server umount lustre-OST0000 complete [ 4154.337701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4154.360611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4154.370927] Lustre: Skipped 2 previous similar messages [ 4156.102726] Lustre: server umount lustre-MDT0000 complete [ 4159.458459] LustreError: 148997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.465456] LustreError: 148997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4160.091881] LustreError: 149145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967502 with bad export cookie 4500787508959983595 [ 4160.096763] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4160.100208] LustreError: 149145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4160.385877] Lustre: server umount lustre-MDT0001 complete [ 4170.488528] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4174.081791] Key type lgssc unregistered [ 4174.338393] LNet: 150958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4174.350486] LNetError: 150958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4174.384974] LNet: Removed LNI 192.168.203.155@tcp [ 4175.730932] Key type .llcrypt unregistered [ 4175.738089] Key type ._llcrypt unregistered [ 4206.881816] Key type ._llcrypt registered [ 4206.883618] Key type .llcrypt registered [ 4207.079425] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4207.882436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4207.892343] alg: No test for adler32 (adler32-zlib) [ 4209.168726] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 4209.400369] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4211.128378] Key type lgssc registered [ 4212.317756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4220.139580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4220.217582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.826693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.945979] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4227.054574] LustreError: 152371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4227.081477] LustreError: 152371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4231.142303] LustreError: 152372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.264731] LustreError: 152371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.086320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4237.313455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4243.011305] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4250.775024] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4255.961836] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4263.880701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4264.382985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4266.495814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4271.726305] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4279.434696] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4282.875540] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4304.567728] Lustre: lustre-MDT0001: evicting 819e70d1-5dbd-4a0c-be39-5a990950712d (at 192.168.203.55@tcp) by administrative request [ 4304.574852] Lustre: Skipped 1 previous similar message [ 4333.025668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4333.039328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.070937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.050796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.056285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4334.058197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.197262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4339.206850] Lustre: Skipped 2 previous similar messages [ 4344.295077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4344.304762] Lustre: Skipped 1 previous similar message [ 4347.360432] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4347.496942] Lustre: server umount lustre-OST0000 complete [ 4354.532305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4354.536446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4354.542516] Lustre: Skipped 1 previous similar message [ 4354.558029] Lustre: Skipped 1 previous similar message [ 4357.781943] Lustre: server umount lustre-MDT0000 complete [ 4359.650052] LustreError: 153102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.697961] LustreError: 153102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4361.596039] LustreError: 152352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967703 with bad export cookie 675457700985126041 [ 4361.605697] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4361.611727] LustreError: 152352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4362.079337] Lustre: server umount lustre-MDT0001 complete [ 4373.402286] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4376.114764] Key type lgssc unregistered [ 4376.504789] LNet: 156452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4376.518230] LNetError: 156452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4376.542758] LNet: Removed LNI 192.168.203.155@tcp [ 4377.431651] Key type .llcrypt unregistered [ 4377.435111] Key type ._llcrypt unregistered [ 4451.477811] Key type ._llcrypt registered [ 4451.485551] Key type .llcrypt registered [ 4451.686368] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4453.354451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4453.443485] alg: No test for adler32 (adler32-zlib) [ 4454.430101] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 4454.583446] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4456.216579] Key type lgssc registered [ 4457.617096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4467.083287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4467.109166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4468.515643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.142400] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4478.950230] LustreError: 158937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.970535] LustreError: 158937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4484.160496] LustreError: 158938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.225806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.623557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4491.153793] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4502.604809] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4509.904050] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4519.106841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4519.693622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.073322] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4529.920294] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 4533.750804] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4538.104603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 4548.579989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.582877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4548.603542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4553.704851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4553.720521] Lustre: Skipped 1 previous similar message [ 4554.795647] Lustre: server umount lustre-OST0000 complete [ 4561.889969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4561.928099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.980632] Lustre: Skipped 1 previous similar message [ 4561.984677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4561.995764] Lustre: Skipped 1 previous similar message [ 4563.938395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4566.010323] Lustre: server umount lustre-MDT0000 complete [ 4569.071976] LustreError: 158937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4569.120526] LustreError: 158937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4570.820813] LustreError: 158920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790967912 with bad export cookie 6871626802399656410 [ 4570.834584] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4571.277632] Lustre: server umount lustre-MDT0001 complete [ 4586.800337] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4589.329530] Key type lgssc unregistered [ 4589.613597] LNet: 162501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4589.624618] LNetError: 162501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4589.663958] LNet: Removed LNI 192.168.203.155@tcp [ 4590.771481] Key type .llcrypt unregistered [ 4590.774118] Key type ._llcrypt unregistered [ 4616.853748] Key type ._llcrypt registered [ 4616.856531] Key type .llcrypt registered [ 4616.927304] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4617.681382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4617.711668] alg: No test for adler32 (adler32-zlib) [ 4618.788637] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 4619.025636] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4620.705816] Key type lgssc registered [ 4621.838180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4636.636786] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 15:06:17 (1790967977) [ 4652.636215] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4662.215968] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4697.877608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4697.942890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4699.290820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4699.370825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4699.580340] Lustre: lustre-MDT0000: new disk, initializing [ 4699.761724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.779630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4703.852577] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4715.084320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4715.172898] Lustre: 166761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4715.275523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4715.282775] Lustre: Skipped 1 previous similar message [ 4715.385354] Lustre: lustre-MDT0001: new disk, initializing [ 4715.468703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4715.516855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4715.535613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4721.273742] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4734.088812] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4739.707434] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4747.961526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4748.391798] Lustre: lustre-OST0000: new disk, initializing [ 4748.396249] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4748.408543] Lustre: 168732:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4748.603439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4754.555243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4754.571230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4754.701222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4756.351967] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4766.219106] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4775.397223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4775.400111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.420156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4775.449207] Lustre: Skipped 1 previous similar message [ 4780.517426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4780.526394] Lustre: Skipped 1 previous similar message [ 4781.599797] Lustre: server umount lustre-OST0000 complete [ 4785.633583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.658955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4785.669541] Lustre: Skipped 2 previous similar messages [ 4790.763658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4790.767228] Lustre: Skipped 1 previous similar message [ 4791.609871] Lustre: server umount lustre-MDT0000 complete [ 4795.238982] LustreError: 166752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790968137 with bad export cookie 10953675190814066937 [ 4795.243694] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4795.253299] LustreError: 166752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4795.614592] Lustre: server umount lustre-MDT0001 complete [ 4806.898866] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4809.875973] Key type lgssc unregistered [ 4810.207761] LNet: 170634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4810.215617] LNetError: 170634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4810.247410] LNet: Removed LNI 192.168.203.155@tcp [ 4811.254508] Key type .llcrypt unregistered [ 4811.257384] Key type ._llcrypt unregistered [ 4838.464943] Key type ._llcrypt registered [ 4838.467633] Key type .llcrypt registered [ 4838.636306] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4839.848575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4839.986254] alg: No test for adler32 (adler32-zlib) [ 4841.214983] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 4841.462637] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4843.152251] Key type lgssc registered [ 4844.303798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4852.739165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4852.763361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4854.253711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4859.094232] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4864.502832] LustreError: 172047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.524197] LustreError: 172047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4867.981176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4868.502522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4873.435173] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4881.513414] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4886.946087] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4893.074542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4893.402734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4899.182216] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4899.568684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4906.097757] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4916.475271] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4916.498207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.521616] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4916.538456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4916.556049] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4917.547861] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4917.567939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.594034] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4917.606594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4917.623711] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4918.643157] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4918.651765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.664954] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4918.668848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4918.677812] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4919.600498] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4920.573542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4921.511453] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4921.515918] Lustre: Skipped 2 previous similar messages [ 4921.531821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.540431] Lustre: Skipped 2 previous similar messages [ 4921.543886] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4921.548893] Lustre: Skipped 2 previous similar messages [ 4921.557501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4921.574895] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4921.593270] Lustre: Skipped 2 previous similar messages [ 4922.545319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4923.380987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4924.255751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4925.063425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4925.951372] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4925.959131] Lustre: Skipped 4 previous similar messages [ 4925.980166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.989939] Lustre: Skipped 4 previous similar messages [ 4926.007170] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4926.012614] Lustre: Skipped 4 previous similar messages [ 4926.016842] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4926.031487] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4926.041349] Lustre: Skipped 4 previous similar messages [ 4926.907640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4927.853615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4928.574401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4929.523798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4930.387127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4931.251587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4932.011961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4932.936249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4933.761626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4934.695594] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4934.703652] Lustre: Skipped 9 previous similar messages [ 4934.719331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4934.726389] Lustre: Skipped 9 previous similar messages [ 4934.729660] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4934.733661] Lustre: Skipped 9 previous similar messages [ 4934.739507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4934.754035] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4934.761650] Lustre: Skipped 9 previous similar messages [ 4935.630956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4936.422602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4937.216606] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4938.055647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4938.864314] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4939.638428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4940.389918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.204812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4941.935047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4942.714145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4943.447852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.217230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4944.995694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4945.823480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4946.675392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4947.591328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4948.441419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4949.612471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4950.689413] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.696059] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4951.703150] Lustre: Skipped 19 previous similar messages [ 4951.744088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4951.751962] Lustre: Skipped 19 previous similar messages [ 4951.756555] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4951.760661] Lustre: Skipped 19 previous similar messages [ 4951.763439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4951.774897] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4951.783029] Lustre: Skipped 19 previous similar messages [ 4952.639384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4953.710166] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4954.786348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4955.817619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4956.781421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4957.716577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4958.837294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4959.864648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4960.986667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4961.820270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4962.956644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.024785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4964.944101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4965.918651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4966.711893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4967.787614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4968.655964] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4969.516718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4970.394829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4971.406674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4972.198528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4973.145777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4974.199777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4975.260385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.134972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.990786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4977.816577] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4978.667673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4979.633305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4980.571878] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4981.600534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4982.710188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4983.476248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.323448] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4984.333409] Lustre: Skipped 33 previous similar messages [ 4984.357311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4984.366533] Lustre: Skipped 33 previous similar messages [ 4984.370593] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4984.376189] Lustre: Skipped 33 previous similar messages [ 4984.382742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4984.394918] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4984.404803] Lustre: Skipped 33 previous similar messages [ 4985.294105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4986.210174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4987.150389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.127713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4988.961552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4989.973837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4990.771020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4991.730983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4992.580996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4993.694493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4994.643391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4995.530547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4996.391612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4997.288520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4998.130780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4999.252026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5000.192184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5001.209681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5002.275281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5003.380201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5004.363835] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5005.477087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5006.640576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5007.746174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5008.597903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5009.618202] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5010.652092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5011.655811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5012.632762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5013.495475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5014.595016] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5015.616185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5016.484250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5017.460252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5018.408761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5019.241249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.096230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5020.920390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5022.014855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.033126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5023.955332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.003967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5025.975960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5027.167388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5028.162112] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5029.206779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5030.290255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5031.174941] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.346222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.061251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5033.960898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5034.879881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5035.828932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5036.710629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5037.844586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5038.932484] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5040.012791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5041.214048] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.240700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.853971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5043.797094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.059958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5045.893361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5046.755882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5047.486430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.757347] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5048.765301] Lustre: Skipped 65 previous similar messages [ 5048.779593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.801901] Lustre: Skipped 65 previous similar messages [ 5048.807307] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5048.812510] Lustre: Skipped 65 previous similar messages [ 5048.820579] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5048.846565] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5048.856924] Lustre: Skipped 65 previous similar messages [ 5049.628106] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5050.899353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5051.840708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5052.855599] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5053.759603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5054.776703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5055.899586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5057.250104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5058.358588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5059.442594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5060.632566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5061.800209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5062.728044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5063.748740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5064.555813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5065.593786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5066.617329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5067.578085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5068.551849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5069.752405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5070.665299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5071.631876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5072.552596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.460523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5074.429833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5075.463561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5076.531301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5077.533302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5078.465850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5079.427489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5080.334364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5081.140475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.016379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5082.778929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5083.648931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5084.568168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5085.381471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5086.199081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.022030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.814352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5088.880673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5089.785373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5090.634705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5091.446487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5092.360758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.181312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5094.101926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5095.044305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5096.021285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.033788] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5097.935976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5098.831070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5099.813797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.644612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5101.567305] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5102.417412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5103.236394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.074609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5104.900912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5105.708514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5106.510889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5107.331492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.164654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.983046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5109.810963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5110.694386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5111.722983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5112.597786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5113.554419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5114.489717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5115.509219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5116.269630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5117.128453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.028652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5118.884986] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5119.778601] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5120.734476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5121.669393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5122.753588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5123.762222] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5124.772816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5125.742872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5126.756325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5127.597428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5128.640023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5129.699321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5130.869526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5131.940502] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5132.967934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.044521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5134.991565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5135.881833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5136.802428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5137.762303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5138.749887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5139.757809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5140.741383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5141.674262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5142.461085] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5143.314225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5144.184605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5145.213871] LustreError: 172067:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5145.227066] LustreError: 172067:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5145.232874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5146.080058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5147.089466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5148.043978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.023042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5149.960862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5150.934253] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5151.920777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5152.902327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5153.932060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5154.808888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5155.759815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5156.785441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5157.783618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5158.771461] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5160.174646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5161.149406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.130282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5162.974310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5163.848477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5164.460679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5165.343823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5166.166153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5167.274429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5168.148151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.068774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5169.859799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5170.933176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5171.912175] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5172.880634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5174.011531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5175.114463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5176.185432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.586557] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5177.601966] Lustre: Skipped 134 previous similar messages [ 5177.628602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.648816] Lustre: Skipped 134 previous similar messages [ 5177.655233] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5177.665595] Lustre: Skipped 134 previous similar messages [ 5177.675515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5177.699481] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5177.710572] Lustre: Skipped 134 previous similar messages [ 5178.617663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5179.593966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5180.781130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5181.734381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5183.318204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5184.677534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5185.876464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5186.867463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5188.121709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5189.178023] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5190.436032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5191.511973] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5192.520425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5193.429726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5194.465818] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5195.508845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.558486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5197.453500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5198.295401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5199.214226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5200.153578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.032916] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5201.885832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5202.744444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5203.712554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5204.616674] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5205.561312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5206.454716] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5207.463658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5208.334508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5209.240316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5210.231237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.233927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5212.212796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5213.079458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.006148] LustreError: 172067:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5214.019981] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5214.021754] LustreError: 172067:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5214.927633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5215.886719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5216.722803] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.514437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5218.294075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5219.157368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5220.089333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.108784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.958953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5222.900968] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5223.806632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5224.721698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5225.530429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5226.396622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5227.259357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5228.273751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5229.270336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5230.229260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5231.208893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5232.115787] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5233.085903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.018121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5234.929684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5235.908782] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5236.793611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5237.952735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5238.912860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5240.024849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5241.023352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.097971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5242.968084] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5243.852556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5244.933205] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5245.984836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5246.942578] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5248.179734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5249.288214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5250.217240] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5251.223352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5252.409585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5253.563655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5254.690610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5255.669927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5256.749386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5257.878046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5258.853337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5259.953860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5260.913398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5261.938565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5262.894950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5263.766554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5264.745693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5265.741751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5266.543213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5267.523785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5268.532030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5269.437989] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5270.503197] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5271.533345] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5272.491958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5273.490588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5274.416262] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5275.391407] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.291736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5277.266364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5278.185532] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5279.282473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.230990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5281.146419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.011462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5282.909647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5283.883976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5284.840455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5285.705617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5286.615399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5287.473201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5288.337444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5289.206200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5290.138142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5291.163574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.047566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5292.995780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5293.730826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5294.743540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5295.579392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.475745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5297.373220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5298.356564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5299.442653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.389114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.288111] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5302.219356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.157380] LustreError: 172067:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5303.170536] LustreError: 172067:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5303.184107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5303.983556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5304.905372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5305.801771] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5306.794557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5307.767126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5308.885956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5309.930225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5310.909442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5311.776664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.698368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5313.503203] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.250644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5314.983278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5315.925494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5316.816846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5317.671064] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5318.583535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5319.522367] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5320.314435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5321.223647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.171343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5323.212187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5324.113984] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5325.144421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.059317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5326.954618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5327.750172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5328.684313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5329.583864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5330.502265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5331.373990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5332.297159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5333.166738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.151587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5334.887544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5335.775958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5336.678590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5337.571127] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5338.436079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5339.270010] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.046138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5340.830970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5341.493211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5342.309727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.095265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5343.980923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5344.933141] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5345.762684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5346.413354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5347.292612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.162857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5348.924562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5349.900128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.047703] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5351.727887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5352.443780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5353.235902] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5354.112379] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.008672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5355.722481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5356.841328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5358.000352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5359.143662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.027567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5360.976041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5361.795763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5362.825053] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5363.797264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5364.906684] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5365.825241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5367.116504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5368.053845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5369.138575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5370.122160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.107648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5371.969571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5372.743912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5373.534616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5374.357801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.097448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5375.869737] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5376.725872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5377.606937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5378.516631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5379.388527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5380.307612] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5381.241231] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5382.204449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5383.252943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5384.090322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5385.056375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5386.189602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5387.141588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.011607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5388.760171] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5389.677169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5390.443273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5391.310676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5392.210726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.098341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5393.894427] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5394.745598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5395.617271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5396.404543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5397.168569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.031756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5398.866617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5399.867295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5400.648211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5401.454209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5402.181547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.067696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5403.937144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5404.844689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5405.553412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5406.503454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5407.352893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5408.232754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.094068] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5409.988697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5410.935702] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5411.890645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5412.822936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5413.750058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5414.701330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5415.661751] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5416.574415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5417.579408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5418.572562] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5419.644547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5420.581662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5421.448168] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5422.350849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5423.411666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5424.244297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.125355] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5425.867677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5426.704342] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5427.582325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5428.454337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5429.395260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5430.353132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5431.352278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.057922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5432.938709] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5433.943795] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5433.950438] Lustre: Skipped 274 previous similar messages [ 5433.969839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.989664] Lustre: Skipped 274 previous similar messages [ 5434.003461] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5434.008553] Lustre: Skipped 274 previous similar messages [ 5434.011830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5434.021994] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5434.028711] Lustre: Skipped 274 previous similar messages [ 5434.677730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5435.607624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5436.567417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5437.499071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5438.516561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5439.346277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.141622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5440.996020] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5441.881522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5442.876281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5443.929892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5444.801439] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5445.755742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5446.595437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5447.456463] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.169226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5448.982644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5449.750860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5450.655717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5451.522363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5452.378512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5453.250701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.031513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5454.986167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5455.831963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5456.700032] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5457.506230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5458.463198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5459.268882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5460.137080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.014364] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5461.857399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5462.764294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5463.709486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5464.452300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5465.294955] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5466.118670] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.048948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5467.866310] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5468.724249] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5469.502572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5470.354260] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.170595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5471.981926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5472.663416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5473.451727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.193186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5474.941476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5475.789605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5476.593789] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5477.394957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5478.243187] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.045907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5479.869447] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5480.729649] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5481.433943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.211210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5482.999761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5483.771826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5484.563963] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5485.287445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.096596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5486.802686] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5487.523076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5488.205177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.053950] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5489.817070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5490.553644] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5491.194209] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.007922] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5492.781063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5493.478033] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5494.208777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.106340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5495.880647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5496.907728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5497.743400] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5498.595584] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5499.369247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5500.248067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.025321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5501.950844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5502.747183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5503.532306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5504.319624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5505.145491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5506.033707] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.017538] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5507.856490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5508.740774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5509.598638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5510.513376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5511.348772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5512.294292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.064797] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5513.982300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5514.911388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5515.835415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5516.665625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5517.513380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5518.321313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5519.130384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.074808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5520.897513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5521.625877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5522.458428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5523.139486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.057920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5524.811625] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5525.802539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5526.696860] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5527.484306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5528.158720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.073244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5529.912199] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5530.739265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5531.487945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5532.451398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5533.545278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5534.719648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5535.861939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.034756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5537.865050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5538.795228] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5539.721506] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.889949] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5542.051799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5543.106284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5544.186594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5545.666343] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5546.792722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5548.149178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5549.114962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5550.219724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.003543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5551.814250] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5552.728298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5553.734396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5554.576588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5555.486953] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5556.299162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5557.248743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5558.112243] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.150891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5559.883386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5560.850938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5561.809360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5562.845544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5563.906034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5564.911161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5565.741074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5566.714480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5567.742723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5568.653201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5569.582259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5570.454799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.422526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5572.450234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.390774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.367745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5575.404884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.516029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.257273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5578.142722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.232496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5580.211130] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.221956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.088312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.012935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.887777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.873944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.755499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.818049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.760890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.760316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.754588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.757524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.508853] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.365198] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.182553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5593.939071] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.639921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.510503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.300676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.018160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.862277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.716028] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.552972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.402534] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.311491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.210913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.177893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.165758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.088813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.942936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.823478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.627410] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.450446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5609.431734] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.383680] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.397385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.398555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5613.318486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.046138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.921373] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.656270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.584917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.362550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.210406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.005785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.799340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.514082] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.485415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.414535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.268575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.069881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.928956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.623377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5626.560444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.324398] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.220637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.135975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.870894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.654395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.458300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.183154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.130766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.287465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.332510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.268903] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.177857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.383917] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.659868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.585626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5641.941212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.034693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5643.958381] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.985350] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.293237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.763414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.131365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.632273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.428609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5652.515602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.432607] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.946521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.818073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.112372] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.053519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.031210] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.405496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.408865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.335306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.025437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.650221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5664.293611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.096772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.801220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.769899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.577482] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.366750] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5669.108971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.140386] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.005946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.914471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.681810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.484850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.328671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.299663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.226429] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.022374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.973760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.827258] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.592241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.460512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.409614] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.427391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.492321] LustreError: 172067:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5683.504169] LustreError: 172067:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5683.507275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.449646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.326976] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.149142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.036167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.915742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.899072] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.703214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.593225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.346242] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.250061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.113757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.091497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.929781] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.927520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.818356] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.732754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.661706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.484391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.471319] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.243923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.156300] LustreError: 172067:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5702.165739] LustreError: 172067:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5702.189634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.338711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.317229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5705.319605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.331904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.614233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.584557] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.589273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.631034] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.656497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5712.481678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.398515] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.532603] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.437764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.383241] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.403942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.363404] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.144478] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.123160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.925786] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.895763] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.983729] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.034881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.064804] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.176159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.183170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.164810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.945200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.015550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.022489] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.010633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.977629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.051526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.944188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.001992] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.009062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.987907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.803453] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.793144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.655663] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.573119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.399290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.337477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.317633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.262159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.132779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.175622] LustreError: 172067:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5747.192507] LustreError: 172067:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5747.198479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.108362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.147912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.229295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.127996] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.941293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.772881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.622974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.405849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.193617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.114285] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.111864] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.995528] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.919062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.843533] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.821374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.642377] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.604887] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.650369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.568617] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.537624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.522330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.561344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.406162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.419881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.342383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.113987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.975249] LustreError: 172067:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5771.987174] LustreError: 172067:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5772.000353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.017662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.869125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5774.729958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.642397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.476548] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.497677] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.397890] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.279882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.226762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.071200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.887711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.687637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.488412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.508743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.303743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.224738] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.035313] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.954207] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5788.825420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.869428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.972753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.938929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.979900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.715855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.652632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.731778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.656434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5797.633914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.570967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.397935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.218650] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.126221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.884778] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.717980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.469118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.380309] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.246561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.076954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.871186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.685088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.485892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.422656] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.256905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.102850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.036767] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.935741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.852229] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.787936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.567092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.421634] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.440450] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5818.420366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.440852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.396882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.400301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.223496] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.357327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.372678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.282444] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.126595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.130158] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.920148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5828.977237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.924174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.910645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.800714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5832.870493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.717632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5834.607779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.549527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.575808] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.366776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.421975] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.319655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5840.266587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.062580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.851266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.147726] LustreError: 173871:0:(ofd_dev.c:1780:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 6161.157775] LustreError: 172067:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 6619.106511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6619.121034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.140292] Lustre: Skipped 449 previous similar messages [ 6619.144033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6620.134161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6620.144905] Lustre: Skipped 1 previous similar message [ 6624.262600] Lustre: server umount lustre-OST0000 complete [ 6629.857357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6629.870364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6634.479790] Lustre: server umount lustre-MDT0000 complete [ 6635.489415] LustreError: 172047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6635.514166] LustreError: 172047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6637.806222] LustreError: 172028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790969979 with bad export cookie 5392865145090343533 [ 6637.819440] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6637.823400] LustreError: 172028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6638.445745] Lustre: server umount lustre-MDT0001 complete [ 6649.719492] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6652.138571] Key type lgssc unregistered [ 6652.361560] LNet: 227255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6652.378437] LNetError: 227255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6652.403736] LNet: Removed LNI 192.168.203.155@tcp [ 6653.211870] Key type .llcrypt unregistered [ 6653.214939] Key type ._llcrypt unregistered [ 6680.814572] Key type ._llcrypt registered [ 6680.816193] Key type .llcrypt registered [ 6680.972579] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6681.906160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6681.938499] alg: No test for adler32 (adler32-zlib) [ 6683.012337] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 6683.226169] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6684.897195] Key type lgssc registered [ 6685.884831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6692.280497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6692.322271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.753907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6696.967594] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6703.075232] LustreError: 228672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6703.094549] LustreError: 228672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6704.036936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.404840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6708.772098] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6716.042476] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6721.050407] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6726.902680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6727.332446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.966237] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6738.628349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49665 to 0x280000401:49696) [ 6740.116159] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6745.585423] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6747.286577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6747.304500] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6747.309200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6747.319310] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6747.320268] LustreError: 228692:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 6749.155411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6749.160310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6749.169852] Lustre: Skipped 1 previous similar message [ 6753.181400] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 6753.188935] Lustre: Skipped 1 previous similar message [ 6754.280878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6754.297788] Lustre: Skipped 1 previous similar message [ 6755.348226] Lustre: server umount lustre-OST0000 complete [ 6767.416097] LustreError: 228692:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 6789.348716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.673161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6795.909461] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6802.520733] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6803.741779] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6805.036544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6805.062981] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6805.064285] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6805.085333] Lustre: Skipped 1 previous similar message [ 6808.668185] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ba086d6c000.ost_server_uuid 50 [ 6809.961914] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ba086d6c000.ost_server_uuid in FULL state after 0 sec [ 6815.206117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.215869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6815.226065] Lustre: Skipped 1 previous similar message [ 6820.331421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6820.351224] Lustre: Skipped 1 previous similar message [ 6820.546896] Lustre: server umount lustre-OST0000 complete [ 6825.444377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6825.482451] Lustre: Skipped 1 previous similar message [ 6830.570777] LustreError: 229399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6830.584126] LustreError: 229399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6830.669414] Lustre: server umount lustre-MDT0000 complete [ 6833.712983] LustreError: 228654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970175 with bad export cookie 2010114587415080458 [ 6833.713649] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6833.739384] LustreError: 228654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6834.037909] Lustre: server umount lustre-MDT0001 complete [ 6845.684402] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6848.673096] Key type lgssc unregistered [ 6849.061993] LNet: 233876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6849.071376] LNetError: 233876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6849.096790] LNet: Removed LNI 192.168.203.155@tcp [ 6849.926627] Key type .llcrypt unregistered [ 6849.933783] Key type ._llcrypt unregistered [ 6880.281581] Key type ._llcrypt registered [ 6880.283928] Key type .llcrypt registered [ 6880.374479] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6881.414534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6881.441356] alg: No test for adler32 (adler32-zlib) [ 6882.509388] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 6882.685493] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6884.336265] Key type lgssc registered [ 6885.304171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6894.350288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6894.386208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.863815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.540321] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6905.316395] LustreError: 235600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6905.339260] LustreError: 235600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6909.162884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6909.503652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6913.689666] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6916.326036] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6922.246815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6922.599086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.990733] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6936.116747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6936.329749] Lustre: lustre-OST0001: new disk, initializing [ 6936.332831] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6936.341063] Lustre: 237906:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6936.410044] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6940.532527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49728) [ 6941.442247] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6943.279923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6943.290331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6943.366720] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6951.634564] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6957.032237] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6969.317983] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.325158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6969.332618] Lustre: Skipped 2 previous similar messages [ 6969.340679] Lustre: Skipped 1 previous similar message [ 6970.167114] Lustre: server umount lustre-MDT0000 complete [ 6974.234283] LustreError: 235580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970316 with bad export cookie 5637056163091515347 [ 6974.237088] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6974.248529] LustreError: 235580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6974.267850] LustreError: 236337:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200001b72:0x1:0x0]: rc = -5 [ 6974.433308] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.437694] LustreError: 235597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.438250] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6974.438254] Lustre: Skipped 2 previous similar messages [ 6974.457639] Lustre: Skipped 2 previous similar messages [ 6974.511763] LustreError: 235597:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6974.800255] Lustre: server umount lustre-MDT0001 complete [ 6981.193952] Lustre: server umount lustre-OST0000 complete [ 6985.191271] Lustre: server umount lustre-OST0001 complete [ 7008.267477] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7019.579482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7020.169565] LustreError: 241017:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7020.355513] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7025.635078] LustreError: 241018:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7026.576980] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7030.764324] LustreError: 241017:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.842357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.414821] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7039.819455] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7042.474579] Lustre: 242161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7048.947769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.722892] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7060.715208] LustreError: 242514:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7060.725683] Lustre: mylustre-OST0000: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 7065.411652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7065.678301] Lustre: mylustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7065.691074] Lustre: Skipped 1 previous similar message [ 7066.685616] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49760) [ 7066.689112] Lustre: mylustre-OST0001: new connection from mylustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3 to 0x2c0000400:33) [ 7072.277581] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7079.889665] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7083.152336] Lustre: 244033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7096.812636] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7096.834707] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7096.851177] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7096.854160] Lustre: Skipped 1 previous similar message [ 7099.586742] Lustre: server umount mylustre-MDT0000 complete [ 7102.435794] LustreError: 241752:0:(ldlm_lib.c:1190:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.471425] LustreError: 241752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7102.924312] LustreError: 244035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970444 with bad export cookie 5637056163091518196 [ 7102.931146] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7102.942134] LustreError: 244035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7103.284639] Lustre: server umount mylustre-MDT0001 complete [ 7117.944549] Lustre: server umount mylustre-OST0000 complete [ 7131.358429] Lustre: server umount mylustre-OST0001 complete [ 7156.006582] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7166.666320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.105723] LustreError: 246342:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7167.157948] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7173.315443] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7183.146909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.374457] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7191.487706] Lustre: 247485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7198.616757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7205.983760] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7208.384834] Lustre: tfs-OST0000: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:161) [ 7212.496956] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49792) [ 7215.050341] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7220.754281] Lustre: tfs-OST0001: new connection from tfs-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:65) [ 7221.924842] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7228.832246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7232.343451] Lustre: 249357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7240.162726] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7240.175357] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.193658] Lustre: Skipped 3 previous similar messages [ 7240.202300] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 7240.220399] Lustre: Skipped 4 previous similar messages [ 7245.678641] Lustre: server umount tfs-MDT0000 complete [ 7246.308350] LustreError: 247078:0:(ldlm_lib.c:1190:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.326987] LustreError: 247078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7249.292772] LustreError: 246318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970591 with bad export cookie 5637056163091520380 [ 7249.296261] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7249.309838] LustreError: 246318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7249.669476] Lustre: server umount tfs-MDT0001 complete [ 7263.245541] Lustre: server umount tfs-OST0000 complete [ 7277.773748] Lustre: server umount tfs-OST0001 complete [ 7302.753151] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7314.955457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.531358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.537500] Lustre: Skipped 3 previous similar messages [ 7320.360289] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7329.802789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.282793] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7338.607685] Lustre: 252750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7345.660333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7352.885363] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7355.334381] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:193) [ 7360.525618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49698 to 0x280000401:49824) [ 7362.317505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.686268] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:97) [ 7369.580176] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7376.980336] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7380.237713] Lustre: 254622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7386.822597] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 15:52:07 (1790970727) [ 7403.498462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.510217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7403.526639] Lustre: Skipped 6 previous similar messages [ 7403.540320] Lustre: Skipped 6 previous similar messages [ 7413.734796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7413.744219] Lustre: Skipped 5 previous similar messages [ 7417.313157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7417.546939] Lustre: server umount lustre-MDT0000 complete [ 7418.853316] LustreError: 251604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.887550] LustreError: 251604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7421.270900] LustreError: 251584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970763 with bad export cookie 5637056163091522298 [ 7421.282742] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.285329] LustreError: 251584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7423.971586] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.991518] Lustre: Skipped 1 previous similar message [ 7427.672218] Lustre: server umount lustre-MDT0001 complete [ 7437.977990] Lustre: server umount lustre-OST0000 complete [ 7456.224152] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7456.473786] Lustre: server umount lustre-OST0001 complete [ 7462.628248] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7471.539392] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7511.324884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7511.685801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7511.733527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7511.912090] Lustre: lustre-MDT0000: new disk, initializing [ 7512.060935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 7512.066873] Lustre: Skipped 3 previous similar messages [ 7512.088933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7517.638649] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7528.864451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7529.030186] Lustre: 257807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7529.142920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7529.149718] Lustre: Skipped 1 previous similar message [ 7529.281121] Lustre: lustre-MDT0001: new disk, initializing [ 7529.431668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7529.446655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7534.707719] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7545.608240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7545.924231] Lustre: lustre-OST0000: new disk, initializing [ 7545.928548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7545.937184] Lustre: 259390:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7547.599511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7547.610921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7547.804731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7553.400860] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7565.318623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7565.593211] Lustre: lustre-OST0001: new disk, initializing [ 7565.598569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7565.625650] Lustre: 260263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7567.337941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7567.369247] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7567.533740] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7573.842089] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7588.268611] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 15:55:28 (1790970928) [ 7606.163408] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 15:55:46 (1790970946) [ 7623.658111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7623.663566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7623.681417] Lustre: Skipped 1 previous similar message [ 7623.706369] Lustre: Skipped 5 previous similar messages [ 7627.799844] Lustre: server umount lustre-OST0000 complete [ 7638.430172] Lustre: server umount lustre-MDT0000 complete [ 7642.349344] LustreError: 258614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970984 with bad export cookie 5637056163091524076 [ 7642.356364] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7642.368042] LustreError: 258614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7648.818171] Lustre: server umount lustre-MDT0001 complete [ 7660.848755] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7736.769777] Lustre: server umount lustre-OST0001 complete [ 7744.984684] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7761.443368] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7799.664851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7800.084919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7800.163190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7800.320223] Lustre: lustre-MDT0000: new disk, initializing [ 7800.428578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.436580] Lustre: Skipped 3 previous similar messages [ 7800.455634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7805.648598] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7817.914350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7818.064537] Lustre: 265119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7818.133456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7818.143889] Lustre: Skipped 1 previous similar message [ 7818.229913] Lustre: lustre-MDT0001: new disk, initializing [ 7818.401490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7818.432550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7823.910803] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7834.837186] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7841.394263] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7848.972471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7849.356698] Lustre: 267090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7850.818584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7850.849406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7850.985956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7857.026907] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7867.126576] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7876.579677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7876.585530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7876.595207] Lustre: Skipped 4 previous similar messages [ 7876.606421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7876.616186] Lustre: Skipped 6 previous similar messages [ 7880.530797] Lustre: server umount lustre-OST0000 complete [ 7884.771746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7899.104130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7899.407069] Lustre: server umount lustre-MDT0000 complete [ 7901.159655] LustreError: 265130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.180241] LustreError: 265130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 7903.528941] LustreError: 265111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971245 with bad export cookie 5637056163091525546 [ 7903.534847] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7903.544783] LustreError: 265111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7903.972712] Lustre: server umount lustre-MDT0001 complete [ 7915.955167] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7919.041486] Key type lgssc unregistered [ 7919.423615] LNet: 268850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7919.431358] LNetError: 268850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7919.461314] LNet: Removed LNI 192.168.203.155@tcp [ 7920.445335] Key type .llcrypt unregistered [ 7920.449788] Key type ._llcrypt unregistered [ 7947.229971] Key type ._llcrypt registered [ 7947.232701] Key type .llcrypt registered [ 7947.309693] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7948.218886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7948.239057] alg: No test for adler32 (adler32-zlib) [ 7949.465861] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 7949.769799] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7951.498368] Key type lgssc registered [ 7952.840144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7961.684893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7961.712361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7963.432662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.640844] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7973.860189] LustreError: 270320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.905674] LustreError: 270320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7977.564371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7977.893791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7982.834448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7990.151484] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7995.794074] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8002.526277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.938960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.051287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 8008.999957] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8016.118154] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8017.155836] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 8022.122136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8022.124541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8025.316160] Lustre: server umount lustre-OST0000 complete [ 8032.739525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8032.757976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8032.767578] Lustre: Skipped 2 previous similar messages [ 8036.039104] Lustre: server umount lustre-MDT0000 complete [ 8037.857820] LustreError: 270317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8037.887147] LustreError: 270317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8039.691369] LustreError: 270300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971381 with bad export cookie 18204395429934272960 [ 8039.695862] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8039.706633] LustreError: 270300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8040.182029] Lustre: server umount lustre-MDT0001 complete [ 8052.651492] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8055.972767] Key type lgssc unregistered [ 8056.403486] LNet: 273901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8056.417926] LNetError: 273901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8056.452888] LNet: Removed LNI 192.168.203.155@tcp [ 8057.494154] Key type .llcrypt unregistered [ 8057.495865] Key type ._llcrypt unregistered [ 8074.874602] Key type ._llcrypt registered [ 8074.880904] Key type .llcrypt registered [ 8074.994195] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8075.950228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8075.968363] alg: No test for adler32 (adler32-zlib) [ 8077.071543] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 8077.316856] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8078.976725] Key type lgssc registered [ 8080.138576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8091.157597] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8091.188454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8106.785837] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8111.904319] LustreError: 275093:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 8138.593834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.751252] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8153.735404] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8155.891049] Lustre: server umount lustre-OST0000 complete [ 8172.205845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8174.988883] Key type lgssc unregistered [ 8175.336856] LNet: 276723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8175.367297] LNetError: 276723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8175.386786] LNet: Removed LNI 192.168.203.155@tcp [ 8176.496278] Key type .llcrypt unregistered [ 8176.500314] Key type ._llcrypt unregistered [ 8215.556135] Key type ._llcrypt registered [ 8215.559820] Key type .llcrypt registered [ 8215.704500] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8216.655691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8216.820189] alg: No test for adler32 (adler32-zlib) [ 8217.916826] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 8218.124604] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8219.785329] Key type lgssc registered [ 8220.627814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8232.355851] loop0: detected capacity change from 0 to 204800000 [ 8232.589939] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8232.608200] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8233.035199] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8233.862343] loop0: detected capacity change from 0 to 204800000 [ 8234.080368] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8234.097688] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8234.426266] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8235.306621] loop0: detected capacity change from 0 to 204800000 [ 8235.506792] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8235.526022] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8235.708757] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8236.784368] loop0: detected capacity change from 0 to 204800000 [ 8236.944296] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8237.855263] loop0: detected capacity change from 0 to 204800000 [ 8237.903764] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8241.678403] loop0: detected capacity change from 0 to 204800000 [ 8241.729270] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8245.718611] loop0: detected capacity change from 0 to 204800000 [ 8245.773386] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8249.975266] loop0: detected capacity change from 0 to 204800000 [ 8250.001427] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 8254.349902] loop0: detected capacity change from 0 to 204800000 [ 8254.518817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8254.556397] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8254.641232] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8254.648728] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8254.658141] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8254.667416] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8254.685037] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 8254.700928] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 8254.711684] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 8254.719906] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 8254.728632] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 8254.740371] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 8254.750192] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 8254.760163] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 8254.769819] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 8254.779144] Lustre: 280134:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 8260.409790] Lustre: server umount MGS complete [ 8261.382993] loop0: detected capacity change from 0 to 204800000 [ 8261.512821] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.339765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8262.380500] Lustre: lustre-MDT0000: Aborting client recovery [ 8263.329987] loop1: detected capacity change from 0 to 204800000 [ 8263.493428] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.550845] Lustre: 280567:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8263.568376] Lustre: 280567:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8263.585812] Lustre: 280567:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8263.599368] Lustre: 280567:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 8263.608547] Lustre: 280567:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 8263.617408] Lustre: 280567:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 8263.633274] Lustre: 280567:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8263.665733] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8263.843503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8263.893816] Lustre: lustre-MDT0001: Aborting client recovery [ 8264.816992] loop2: detected capacity change from 0 to 204800000 [ 8264.961042] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8265.017888] Lustre: 280645:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8265.031860] Lustre: 280645:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8265.044732] Lustre: 280645:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 8265.056511] Lustre: 280645:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8265.067818] Lustre: 280645:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 8265.077320] Lustre: 280645:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 8265.085816] Lustre: 280645:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8265.232602] Lustre: Found index 0 for lustre-OST0000, updating log [ 8265.306461] Lustre: 280663:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8265.359838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8265.384388] Lustre: lustre-OST0000: Aborting recovery [ 8266.446035] loop3: detected capacity change from 0 to 204800000 [ 8266.579154] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8266.710173] Lustre: 280728:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 8266.733350] Lustre: 280728:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 8266.747602] Lustre: 280728:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 8266.768049] Lustre: 280728:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 8266.786055] Lustre: 280728:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 8266.794883] Lustre: 280728:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 8266.803380] Lustre: 280728:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 8266.826905] Lustre: Found index 1 for lustre-OST0001, updating log [ 8266.854798] Lustre: 280734:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8266.913366] Lustre: lustre-OST0001: Aborting recovery [ 8272.039428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8272.041734] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 8272.057611] LustreError: 280511:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 8272.087074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 8272.103415] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 8273.064356] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 8273.066302] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 8276.053225] LustreError: 280478:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 649us: rc = -2 [ 8276.573469] LustreError: 280476:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 1862us: rc = -2 [ 8276.608666] LustreError: 280476:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 15 previous similar messages [ 8289.769447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8289.776846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.790802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8293.857936] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8293.864112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8293.872035] Lustre: Skipped 1 previous similar message [ 8293.896929] Lustre: Skipped 3 previous similar messages [ 8295.903647] Lustre: server umount lustre-MDT0000 complete [ 8297.004744] LustreError: 280462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971638 with bad export cookie 4243169808920937648 [ 8297.013667] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8297.025983] LustreError: 280462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8298.978772] LustreError: 280475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8298.979241] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.005389] LustreError: 280475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8299.015537] Lustre: Skipped 1 previous similar message [ 8299.025631] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8299.029709] Lustre: Skipped 1 previous similar message [ 8303.019645] LustreError: 280994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8303.020231] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8303.043121] LustreError: 280994:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8303.048662] Lustre: Skipped 1 previous similar message [ 8303.539727] Lustre: server umount lustre-MDT0001 complete [ 8304.415310] LustreError: 281176:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000400:0x2:0x0].0x0 (000000000911722c) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8310.556490] Lustre: server umount lustre-OST0000 complete [ 8317.567494] Lustre: server umount lustre-OST0001 complete [ 8327.483958] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 16:07:48 (1790971668) [ 8354.762609] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8364.406815] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8404.110155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8404.464306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8404.487986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8404.573517] Lustre: lustre-MDT0000: new disk, initializing [ 8404.742603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8404.751245] Lustre: Skipped 1 previous similar message [ 8404.788631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8409.576219] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8421.004161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8421.141342] Lustre: 284801:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8421.214735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8421.225035] Lustre: Skipped 1 previous similar message [ 8421.326568] Lustre: lustre-MDT0001: new disk, initializing [ 8421.475521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8421.542443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8421.548046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8425.890658] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8435.255430] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8440.669925] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8447.722421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8448.022685] Lustre: lustre-OST0000: new disk, initializing [ 8448.026165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8448.034276] Lustre: 286773:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8448.113985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8454.881982] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8457.305387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8457.320344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8457.433206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8464.041767] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8469.191441] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8481.769675] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8486.985705] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8492.291652] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [ 8529.385538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8529.385688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8529.399226] Lustre: Skipped 1 previous similar message [ 8529.409856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8529.414458] Lustre: Skipped 1 previous similar message [ 8534.494914] Lustre: server umount lustre-OST0000 complete [ 8539.105299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8539.118145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.142039] Lustre: Skipped 1 previous similar message [ 8539.146982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8544.521285] Lustre: server umount lustre-MDT0000 complete [ 8544.741432] LustreError: 284814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8544.774058] LustreError: 284814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8548.995821] LustreError: 286783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971890 with bad export cookie 4243169808920940350 [ 8549.004734] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8549.005214] LustreError: 286783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8549.560592] Lustre: server umount lustre-MDT0001 complete [ 8558.666917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8563.946940] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8569.332684] Lustre: server umount MGS complete [ 8580.985698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.483756] LustreError: 290684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8581.513994] LustreError: 290696:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8581.596268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8586.210419] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8586.723857] LustreError: 290683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.639565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.091766] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8608.756841] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8614.015863] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8621.674677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8622.186599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8622.197310] Lustre: Skipped 1 previous similar message [ 8628.118517] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8636.939094] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8665.823871] Lustre: server umount lustre-OST0000 complete [ 8671.212759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.227716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8671.239966] Lustre: Skipped 2 previous similar messages [ 8671.259320] Lustre: Skipped 2 previous similar messages [ 8679.528970] Lustre: server umount lustre-MDT0000 complete [ 8681.443805] LustreError: 292775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8681.478195] LustreError: 292775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8682.464367] Lustre: 278096:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790972008/real 1790972008] req@000000005a119e53 x1877969794679552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790972024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8682.495298] Lustre: 278096:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8683.779397] LustreError: 290665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972025 with bad export cookie 4243169808920942177 [ 8683.783458] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8683.788974] LustreError: 290665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8684.287407] Lustre: server umount lustre-MDT0001 complete [ 8695.687331] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8698.870710] Key type lgssc unregistered [ 8699.114869] LNet: 294626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8699.130785] LNetError: 294626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8699.145502] LNet: Removed LNI 192.168.203.155@tcp [ 8700.224728] Key type .llcrypt unregistered [ 8700.227944] Key type ._llcrypt unregistered [ 8734.005570] Key type ._llcrypt registered [ 8734.008247] Key type .llcrypt registered [ 8734.101814] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8750.015904] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8750.991323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8751.057186] alg: No test for adler32 (adler32-zlib) [ 8752.291287] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 8752.534703] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8754.217887] Key type lgssc registered [ 8755.295865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8788.840982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8788.886514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8790.144966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8790.172431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8790.296622] Lustre: lustre-MDT0000: new disk, initializing [ 8790.420893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8790.455231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8795.144427] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8805.386464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8805.507388] Lustre: 298578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8805.563358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8805.567953] Lustre: Skipped 1 previous similar message [ 8805.675857] Lustre: lustre-MDT0001: new disk, initializing [ 8805.838067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8805.925121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8805.940566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8811.490343] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8822.894916] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8828.896701] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8836.022483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8836.484266] Lustre: lustre-OST0000: new disk, initializing [ 8836.487113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8836.492187] Lustre: 300548:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8836.611988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8842.321226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8842.337329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8842.479535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8844.387903] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8854.070047] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8860.656493] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 8869.902714] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 8877.985334] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 8934.882880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8934.888372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8934.896349] LustreError: Skipped 1 previous similar message [ 8934.902675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8934.918025] Lustre: Skipped 1 previous similar message [ 8934.951035] Lustre: Skipped 1 previous similar message [ 8936.541533] Lustre: server umount lustre-OST0000 complete [ 8944.105901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8944.122800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.148810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8945.124993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8946.701117] Lustre: server umount lustre-MDT0000 complete [ 8950.398669] LustreError: 298572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972292 with bad export cookie 14357656269366516282 [ 8950.414925] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8950.421550] LustreError: 298572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8950.863724] Lustre: server umount lustre-MDT0001 complete [ 8958.203333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8962.651514] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8967.377098] Lustre: server umount MGS complete [ 8978.213726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8978.563445] LustreError: 304415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.578624] LustreError: 304415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8978.590615] LustreError: 304428:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 8978.652904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.456288] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8984.035945] LustreError: 304416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8988.137030] LustreError: 304415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8989.724284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.079079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8994.589670] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9001.240370] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9006.963739] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9013.286518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9013.625230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.679294] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9027.287349] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9055.713670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9055.719717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9055.731701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9055.734412] Lustre: Skipped 2 previous similar messages [ 9059.812597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9059.817250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9059.817370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.834993] Lustre: Skipped 1 previous similar message [ 9061.148628] Lustre: server umount lustre-OST0000 complete [ 9064.935869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9064.940325] Lustre: Skipped 1 previous similar message [ 9070.622986] Lustre: server umount lustre-MDT0000 complete [ 9074.025788] LustreError: 304395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972416 with bad export cookie 14357656269366518137 [ 9074.030310] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9074.034325] LustreError: 304395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9074.295387] Lustre: server umount lustre-MDT0001 complete [ 9083.217369] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9085.573622] Key type lgssc unregistered [ 9085.829687] LNet: 308358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9085.845500] LNetError: 308358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9085.859986] LNet: Removed LNI 192.168.203.155@tcp [ 9086.579476] Key type .llcrypt unregistered [ 9086.582653] Key type ._llcrypt unregistered [ 9104.145339] Key type ._llcrypt registered [ 9104.148535] Key type .llcrypt registered [ 9104.251297] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9115.781749] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9116.803068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9116.846917] alg: No test for adler32 (adler32-zlib) [ 9117.882365] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [ 9118.100189] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9119.752555] Key type lgssc registered [ 9120.798788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9140.520600] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9153.082924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9153.123649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.353454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9154.374668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9154.446888] Lustre: lustre-MDT0000: new disk, initializing [ 9154.519728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.545083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9158.632812] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9168.315034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9168.570657] Lustre: lustre-OST0000: new disk, initializing [ 9168.577099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9168.584646] Lustre: Skipped 1 previous similar message [ 9168.595522] Lustre: 312383:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9168.682977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9174.030326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 9174.037212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 9174.070602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 9174.648853] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10025.987971] Lustre: server umount lustre-MDT0000 complete [10029.735718] LustreError: 311569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973371 with bad export cookie 3132147937872086231 [10029.751293] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10039.483445] Lustre: server umount lustre-OST0000 complete [10068.436468] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10078.441358] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10113.531398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10113.765382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10113.800300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10113.865780] Lustre: lustre-MDT0000: new disk, initializing [10113.928064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10113.949114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10118.538888] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10129.403971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10129.518818] Lustre: 316721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10129.563765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10129.567690] Lustre: Skipped 1 previous similar message [10129.624963] Lustre: lustre-MDT0001: new disk, initializing [10129.700488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10129.734573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10129.748983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10134.800925] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10144.262148] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10150.453387] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10157.446763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10157.692720] Lustre: lustre-OST0000: new disk, initializing [10157.697381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10157.704374] Lustre: 318691:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10157.811440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10164.311334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10164.320834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10164.413294] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10164.468993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10174.616441] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10179.818207] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10180.059281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10185.397467] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10185.655040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10190.306223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10190.317863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10190.333645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10190.347609] Lustre: Skipped 1 previous similar message [10195.426548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10195.438138] Lustre: Skipped 1 previous similar message [10200.552734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10200.559965] Lustre: Skipped 1 previous similar message [10202.080364] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10202.445896] Lustre: server umount lustre-OST0000 complete [10206.698409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10206.707973] LustreError: Skipped 1 previous similar message [10206.714716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10206.745248] Lustre: Skipped 1 previous similar message [10206.761502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10210.791330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10210.794176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10210.812427] Lustre: Skipped 1 previous similar message [10214.898910] Lustre: server umount lustre-MDT0000 complete [10218.992700] LustreError: 316714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973560 with bad export cookie 3132147937875027050 [10218.996948] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10219.001479] LustreError: 316714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10219.552953] Lustre: server umount lustre-MDT0001 complete [10223.541654] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [10225.219590] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 16:39:26 (1790973566) [10232.553493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10233.178492] LustreError: 321007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10233.195985] LustreError: 321007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10233.262365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10238.397605] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10238.434074] LustreError: 321008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10243.561520] LustreError: 321007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10247.590455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10248.003947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10253.137411] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10261.001380] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10267.480862] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10275.766231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10276.008254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10284.284958] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10297.747368] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10307.159288] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10307.486303] Lustre: lustre-OST0001: new disk, initializing [10307.489846] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10307.497515] Lustre: 323847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10307.667797] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10315.029987] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10315.334360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10315.342444] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10315.409274] LustreError: 324228:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10315.416027] LustreError: 324228:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10316.463149] LustreError: 324222:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10316.497424] LustreError: 324222:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10317.546192] LustreError: 324228:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10317.558825] LustreError: 324228:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [10319.601037] LustreError: 324222:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10319.609721] LustreError: 324222:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [10323.745907] LustreError: 324228:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [10323.778662] LustreError: 324228:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [10326.565416] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10326.904520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10332.080727] LustreError: 324222:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [10332.105749] LustreError: 324222:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 16 previous similar messages [10333.515862] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8ba08482b800.ost_server_uuid 50 [10335.259488] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ba08482b800.ost_server_uuid in FULL state after 0 sec [10341.895839] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ba08482b800.ost_server_uuid 50 [10343.581878] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ba08482b800.ost_server_uuid in IDLE state after 0 sec [10345.551516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10360.803360] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10360.805140] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10360.823298] LustreError: Skipped 1 previous similar message [10360.833241] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10360.840938] Lustre: Skipped 1 previous similar message [10360.857261] Lustre: Skipped 1 previous similar message [10373.088384] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10373.305034] Lustre: server umount lustre-OST0001 complete [10377.107529] LustreError: 322830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.747870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10379.762468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10379.771443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10379.774356] Lustre: Skipped 9 previous similar messages [10382.310817] LustreError: 322831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.333385] LustreError: 322831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10385.356856] Lustre: server umount lustre-OST0000 complete [10391.521906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10391.525699] LustreError: Skipped 1 previous similar message [10391.546728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.582895] Lustre: Skipped 1 previous similar message [10395.902469] Lustre: server umount lustre-MDT0000 complete [10397.667669] LustreError: 321008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.688988] LustreError: 321008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10399.931748] LustreError: 320989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973741 with bad export cookie 3132147937875027932 [10399.934456] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10399.955458] LustreError: 320989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10400.447864] Lustre: server umount lustre-MDT0001 complete [10412.105577] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10415.074249] Key type lgssc unregistered [10415.493677] LNet: 326538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10415.506944] LNetError: 326538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10415.531162] LNet: Removed LNI 192.168.203.155@tcp [10416.406200] Key type .llcrypt unregistered [10416.411120] Key type ._llcrypt unregistered [10449.328702] Key type ._llcrypt registered [10449.335123] Key type .llcrypt registered [10449.490565] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10451.090422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10451.115204] alg: No test for adler32 (adler32-zlib) [10452.308333] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [10452.563762] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10454.275719] Key type lgssc registered [10455.646858] Lustre: Echo OBD driver; http://www.lustre.org/ [10463.887768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10463.903840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.329782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10470.073053] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10475.492499] LustreError: 327960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10475.515338] LustreError: 327960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10478.243691] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10480.627668] LustreError: 327961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10485.733816] LustreError: 327960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10486.120756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10486.532131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10491.635521] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10501.878218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10502.503793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10508.799936] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10510.696341] LustreError: 329640:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10510.717187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [10516.732246] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10516.966944] LustreError: 329641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10517.005931] LustreError: 329641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10523.597856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10523.868309] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10529.278183] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [10529.351857] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10537.925330] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10544.873945] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [10545.135523] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10652.132509] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10652.140427] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10652.144129] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10652.175041] Lustre: Skipped 1 previous similar message [10652.194300] Lustre: Skipped 1 previous similar message [10654.360603] Lustre: server umount lustre-OST0001 complete [10657.250106] LustreError: 330310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10657.295752] LustreError: 330310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10662.370075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10662.375797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10662.401558] Lustre: Skipped 1 previous similar message [10662.415481] Lustre: Skipped 1 previous similar message [10667.495829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10667.505521] Lustre: Skipped 1 previous similar message [10672.614124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10672.628591] Lustre: Skipped 1 previous similar message [10675.170747] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10675.338471] Lustre: server umount lustre-OST0000 complete [10681.313345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10681.321060] LustreError: Skipped 1 previous similar message [10681.330408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10681.351859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10685.926996] Lustre: server umount lustre-MDT0000 complete [10687.982958] LustreError: 327961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10688.021880] LustreError: 327961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10689.530418] LustreError: 331645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974031 with bad export cookie 18188962029565758580 [10689.547872] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10689.548066] LustreError: 331645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10690.095260] Lustre: server umount lustre-MDT0001 complete [10702.269821] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10705.431312] Key type lgssc unregistered [10705.817797] LNet: 332904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10705.837366] LNetError: 332904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10705.859591] LNet: Removed LNI 192.168.203.155@tcp [10706.984624] Key type .llcrypt unregistered [10706.987232] Key type ._llcrypt unregistered [10753.237257] Key type ._llcrypt registered [10753.238976] Key type .llcrypt registered [10753.320639] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10754.421653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10754.487848] alg: No test for adler32 (adler32-zlib) [10755.455663] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [10755.661673] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10757.361515] Key type lgssc registered [10758.539862] Lustre: Echo OBD driver; http://www.lustre.org/ [10767.950828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10767.961890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.432359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10773.659599] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10779.642409] LustreError: 335193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.657824] LustreError: 335193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10782.575492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10783.343185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10789.152510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10792.675889] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10800.213409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10800.611459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10807.234352] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10808.807819] LustreError: 336682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.929462] LustreError: 336682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10814.981757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [10815.072757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10815.302406] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10820.603343] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [10821.691361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10829.445818] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10834.798120] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10846.181084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10846.184652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10846.202296] Lustre: Skipped 1 previous similar message [10846.217936] Lustre: Skipped 3 previous similar messages [10851.303795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10851.314504] Lustre: Skipped 3 previous similar messages [10852.340554] Lustre: server umount lustre-MDT0000 complete [10856.024295] LustreError: 338572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974198 with bad export cookie 11243887166711229382 [10856.031227] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10856.040292] LustreError: 338572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10856.431773] Lustre: server umount lustre-MDT0001 complete [10870.654336] Lustre: server umount lustre-OST0000 complete [10872.800436] Lustre: 334455:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790974198/real 1790974198] req@ffff91e103a6f100 x1877972454797184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790974214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10872.832349] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10872.845604] Lustre: Skipped 2 previous similar messages [10874.689513] Lustre: server umount lustre-OST0001 complete [10891.738691] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10901.480775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10901.942632] LustreError: 340457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10901.964871] LustreError: 340457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10902.034914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10906.140065] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10907.109863] LustreError: 340458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10907.134867] LustreError: 340458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10914.390546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10914.733613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10919.252357] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10921.744170] Lustre: 341597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10927.499242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10934.510434] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10940.070365] LustreError: 341950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10940.099434] LustreError: 341950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10943.280377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10943.520071] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10943.529407] Lustre: Skipped 1 previous similar message [10946.635864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [10946.645925] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [10949.848834] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10958.073437] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10961.658387] Lustre: 343470:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10966.002425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10966.019128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10966.040174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10972.136936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10972.158216] Lustre: Skipped 7 previous similar messages [10972.448866] Lustre: server umount lustre-MDT0000 complete [10975.872172] LustreError: 340437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974317 with bad export cookie 11243887166711231083 [10975.888391] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10976.495180] Lustre: server umount lustre-MDT0001 complete [10990.923594] Lustre: server umount lustre-OST0000 complete [10993.376132] Lustre: 334457:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790974319/real 1790974319] req@ffff91e1031ac700 x1877972454845440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790974335 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10993.417720] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10993.429112] Lustre: Skipped 3 previous similar messages [10994.737910] Lustre: server umount lustre-OST0001 complete [10998.345938] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [11000.163387] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [11002.408796] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 16:52:22 (1790974342) [11025.801305] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11036.656960] loop0: detected capacity change from 0 to 1099511627776 [11037.216548] SGI XFS with ACLs, security attributes, quota, no debug enabled [11037.223886] *** VALIDATE xfs *** [11037.236443] XFS (loop0): Mounting V5 Filesystem [11037.255435] XFS (loop0): Ending clean mount [11038.275706] loop1: detected capacity change from 0 to 18691697672192 [11152.007886] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [11163.611611] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11206.719253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11207.031553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11207.074505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11207.158928] Lustre: lustre-MDT0000: new disk, initializing [11207.240385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11207.256922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11211.934531] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11224.208590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.420739] Lustre: 348606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11224.531583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11224.535868] Lustre: Skipped 1 previous similar message [11224.669230] Lustre: lustre-MDT0001: new disk, initializing [11224.915778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11224.941533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11230.668207] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11241.384469] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11247.670408] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11254.524030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11254.911924] Lustre: lustre-OST0000: new disk, initializing [11254.917235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11254.924480] Lustre: 350577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11256.942618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11256.964951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11257.108444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11262.022448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11271.839797] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11276.968057] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11277.222588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11281.853218] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11282.100887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11288.035262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.041993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11288.042502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11288.052786] Lustre: Skipped 1 previous similar message [11298.275975] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11298.278064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11298.291019] Lustre: Skipped 4 previous similar messages [11298.511938] Lustre: server umount lustre-OST0000 complete [11303.394958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11303.436021] Lustre: Skipped 1 previous similar message [11308.524765] LustreError: 349452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.614672] Lustre: server umount lustre-MDT0000 complete [11312.642247] LustreError: 348599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974654 with bad export cookie 11243887166711232637 [11312.647253] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11312.658652] LustreError: 348599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11313.242693] Lustre: server umount lustre-MDT0001 complete [11322.763968] XFS (loop0): Unmounting Filesystem [11327.079565] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 16:57:48 (1790974668) [11334.035778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11334.673498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11334.690251] Lustre: Skipped 2 previous similar messages [11338.745911] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11344.878316] LustreError: 353137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.900461] LustreError: 353137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11347.006206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11350.881594] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11358.899639] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11364.033425] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11371.077859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11379.763908] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11388.640247] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11400.673872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11400.677869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11400.691973] Lustre: Skipped 1 previous similar message [11400.714137] Lustre: Skipped 3 previous similar messages [11405.364218] Lustre: server umount lustre-OST0000 complete [11416.034679] LustreError: 355793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11416.054437] LustreError: 355793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11416.149976] Lustre: server umount lustre-MDT0000 complete [11419.527234] LustreError: 353118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974761 with bad export cookie 11243887166711233512 [11419.527928] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11419.557758] LustreError: 353118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11419.883644] Lustre: server umount lustre-MDT0001 complete [11430.634032] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11433.663811] Key type lgssc unregistered [11434.040087] LNet: 356767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11434.045653] LNetError: 356767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11434.075743] LNet: Removed LNI 192.168.203.155@tcp [11435.128180] Key type .llcrypt unregistered [11435.130447] Key type ._llcrypt unregistered [11460.562307] Key type ._llcrypt registered [11460.566451] Key type .llcrypt registered [11460.670914] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11461.734096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11461.762466] alg: No test for adler32 (adler32-zlib) [11462.884066] Lustre: Lustre: Build Version: 2.17.58_4_g9ae3af7 [11463.185323] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11464.931237] Key type lgssc registered [11466.088079] Lustre: Echo OBD driver; http://www.lustre.org/ [11473.351958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11473.384297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11475.024508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11479.865449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11484.138199] LustreError: 358328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11484.161343] LustreError: 358328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11488.751370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11489.139874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11493.549363] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11501.644754] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11507.673635] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11514.307028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11514.880537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11521.254421] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11527.830448] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11534.746181] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [11535.332829] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11535.338911] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11535.356567] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11539.843229] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [11539.852352] Lustre: Skipped 1 previous similar message [11540.550437] Lustre: server umount lustre-MDT0001 complete [11544.969126] LustreError: 358325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11550.077047] LustreError: 358324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11550.100185] LustreError: 358324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11550.299218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11550.393184] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11550.397480] Lustre: Found index 1 for lustre-MDT0001, updating log [11550.444413] Lustre: 358318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11550.710431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11555.144627] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11555.865738] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11559.426846] LustreError: 361942:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11559.436412] LustreError: 361942:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11559.449518] LustreError: 358322:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11559.560555] LustreError: 361945:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -17 [11559.568797] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [11559.568797] [11862.444413] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [11862.453775] Lustre: Skipped 2 previous similar messages [11863.017196] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11863.021690] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.049273] Lustre: Skipped 1 previous similar message [11867.512778] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [11867.526441] Lustre: Skipped 2 previous similar messages [11874.272173] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11874.428542] Lustre: server umount lustre-MDT0001 complete [11877.758174] LustreError: 360167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11882.873217] LustreError: 360167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11882.905295] LustreError: 360167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11885.783698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11885.910600] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11885.918951] Lustre: Found index 1 for lustre-MDT0001, updating log [11886.004268] Lustre: 358318:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11886.321247] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [11891.572564] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11891.696561] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11894.780706] LustreError: 370568:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [11894.798413] LustreError: 370568:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [11894.815930] LustreError: 358322:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [11894.826425] LustreError: 358322:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [11894.900263] LustreError: 370571:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [11894.908328] LustreError: 370571:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [11894.911918] LustreError: 370571:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -17 [11894.918750] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.203.155@tcp [11894.918750] [12198.884165] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12198.897268] Lustre: Skipped 1 previous similar message [12198.919597] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12198.922442] Lustre: Skipped 5 previous similar messages [12205.118915] Lustre: server umount lustre-MDT0001 complete [12205.443336] LustreError: 358324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12205.473520] LustreError: 358324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12217.052431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12217.195986] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12217.204254] Lustre: Found index 1 for lustre-MDT0001, updating log [12217.254789] Lustre: 358317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12217.635660] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [12222.855364] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12222.968523] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12227.083182] LustreError: 379190:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [12227.114350] LustreError: 379190:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [12227.134774] LustreError: 358322:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [12227.141119] LustreError: 358322:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [12227.247477] LustreError: 379193:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [12227.267509] LustreError: 379193:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.155@tcp: cfg command failed: rc = -114 [12227.286267] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [12227.286267] [12545.514656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12545.518831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12545.524133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12545.524141] Lustre: Skipped 3 previous similar messages [12545.533210] Lustre: Skipped 3 previous similar messages [12550.833140] Lustre: server umount lustre-MDT0000 complete [12554.702344] LustreError: 360968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975896 with bad export cookie 6633440314488157032 [12554.715200] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12554.717907] LustreError: 360968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12555.748025] LustreError: 358323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12555.752434] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12555.764804] LustreError: 358323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12569.056206] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12569.557923] Lustre: server umount lustre-MDT0001 complete [12574.393850] Lustre: server umount lustre-OST0000 complete [12584.914546] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12594.757230] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12630.982339] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 17:19:31 (1790975971) [12637.723737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12638.013739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12638.062503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12638.152238] Lustre: lustre-MDT0000: new disk, initializing [12638.233528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12638.254981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12642.795861] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12653.255570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12653.500325] Lustre: 390022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12653.561341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12653.566893] Lustre: Skipped 1 previous similar message [12653.635232] Lustre: lustre-MDT0001: new disk, initializing [12653.760751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12653.802848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12653.813507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12658.889996] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12669.800094] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12678.274378] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12686.466887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12686.852497] Lustre: lustre-OST0000: new disk, initializing [12686.857804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12686.860666] Lustre: 391991:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12686.949404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12695.284458] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12696.151229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12696.169675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12696.330835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12705.541457] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12714.984298] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12714.990270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12715.005695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12715.008252] Lustre: Skipped 8 previous similar messages [12715.457908] Lustre: server umount lustre-MDT0000 complete [12717.034829] LustreError: 390033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12717.059300] LustreError: 390033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12720.004275] LustreError: 390015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976061 with bad export cookie 6633440314488158985 [12720.012617] LustreError: 390829:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [12720.031808] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12720.034300] LustreError: 390015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12720.728508] Lustre: server umount lustre-MDT0001 complete [12735.663762] Lustre: server umount lustre-OST0000 complete [12765.506994] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12777.272756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12811.402362] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12573 sec ========= 17:22:32 (1790976152) [12813.237867] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:22:33 (1790976153) === [12816.589378] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:22:37 (1790976157) === [12853.777298] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12856.555865] Key type lgssc unregistered [12856.837733] LNet: 398357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12856.851803] LNetError: 398357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12856.878800] LNet: Removed LNI 192.168.203.155@tcp [12857.813724] Key type .llcrypt unregistered [12857.823062] Key type ._llcrypt unregistered