[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 510620374 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003296] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008439] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011137] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.014029] SELinux: Initializing. [ 0.015081] *** VALIDATE selinux *** [ 0.024221] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029093] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032128] *** VALIDATE tmpfs *** [ 0.033588] *** VALIDATE proc *** [ 0.034269] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038174] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044759] debug: unmapping init [mem 0xffffffffa4459000-0xffffffffa4460fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048029] ... version: 2 [ 0.049018] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055327] rcu: Hierarchical SRCU implementation. [ 0.057552] smp: Bringing up secondary CPUs ... [ 0.058664] x86: Booting SMP configuration: [ 0.059031] .... node #0, CPUs: #1 #2 #3 [ 0.062631] smp: Brought up 1 node, 4 CPUs [ 0.063976] smpboot: Max logical packages: 1 [ 0.064014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134998] node 0 deferred pages initialised in 68ms [ 0.138090] devtmpfs: initialized [ 0.139171] x86/mm: Memory block size: 128MB [ 0.141649] gcov: version magic: 0x41383552 [ 0.143422] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148305] pinctrl core: initialized pinctrl subsystem [ 0.149226] [ 0.149871] ************************************************************* [ 0.150021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151014] ** ** [ 0.152012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153027] ** ** [ 0.154019] ** This means that this kernel is built to expose internal ** [ 0.155015] ** IOMMU data structures, which may compromise security on ** [ 0.156021] ** your system. ** [ 0.157013] ** ** [ 0.158019] ** If you see this message and you are not debugging the ** [ 0.159015] ** kernel, report this immediately to your vendor! ** [ 0.160018] ** ** [ 0.161015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162019] ************************************************************* [ 0.163923] NET: Registered protocol family 16 [ 0.164505] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.167474] cpuidle: using governor menu [ 0.169148] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171483] PCI: Using configuration type 1 for base access [ 0.173154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.184078] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.189222] cryptd: max_cpu_qlen set to 1000 [ 0.193290] ACPI: Added _OSI(Module Device) [ 0.194016] ACPI: Added _OSI(Processor Device) [ 0.195013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200158] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.202371] ACPI: Interpreter enabled [ 0.203040] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203952] ACPI: Using IOAPIC for interrupt routing [ 0.204092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205302] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.214448] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.215035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.216023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.217083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.219363] acpiphp: Slot [2] registered [ 0.220104] acpiphp: Slot [5] registered [ 0.221109] acpiphp: Slot [6] registered [ 0.222100] acpiphp: Slot [7] registered [ 0.223096] acpiphp: Slot [8] registered [ 0.224087] acpiphp: Slot [9] registered [ 0.225118] acpiphp: Slot [10] registered [ 0.226102] acpiphp: Slot [3] registered [ 0.227078] acpiphp: Slot [4] registered [ 0.228088] acpiphp: Slot [11] registered [ 0.229083] acpiphp: Slot [12] registered [ 0.230074] acpiphp: Slot [13] registered [ 0.231111] acpiphp: Slot [14] registered [ 0.232090] acpiphp: Slot [15] registered [ 0.233082] acpiphp: Slot [16] registered [ 0.234081] acpiphp: Slot [17] registered [ 0.235093] acpiphp: Slot [18] registered [ 0.236102] acpiphp: Slot [19] registered [ 0.237084] acpiphp: Slot [20] registered [ 0.238124] acpiphp: Slot [21] registered [ 0.239085] acpiphp: Slot [22] registered [ 0.240094] acpiphp: Slot [23] registered [ 0.241084] acpiphp: Slot [24] registered [ 0.242081] acpiphp: Slot [25] registered [ 0.243089] acpiphp: Slot [26] registered [ 0.244135] acpiphp: Slot [27] registered [ 0.246222] acpiphp: Slot [28] registered [ 0.248767] acpiphp: Slot [29] registered [ 0.251199] acpiphp: Slot [30] registered [ 0.253157] acpiphp: Slot [31] registered [ 0.255129] PCI host bridge to bus 0000:00 [ 0.257024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.259029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.262024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.265024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.268015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.271019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.272197] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.276000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.279399] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.288941] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.294032] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.296021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.297023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.299021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.301696] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.305089] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.307051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.308801] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.315025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.331019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.337015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.341959] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.352020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.362018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.384017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.393904] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.402016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.409021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.427024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.440572] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.451016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.460021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.487017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.502268] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.509018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.514019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.530019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.539689] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.549019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.581024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.589316] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.598018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.603021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.621019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.632154] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.636380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.641410] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.643391] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.645252] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.649200] iommu: Default domain type: Passthrough [ 0.651504] SCSI subsystem initialized [ 0.653200] ACPI: bus type USB registered [ 0.654106] usbcore: registered new interface driver usbfs [ 0.656087] usbcore: registered new interface driver hub [ 0.658101] usbcore: registered new device driver usb [ 0.659224] pps_core: LinuxPPS API ver. 1 registered [ 0.661018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.665078] PTP clock support registered [ 0.667118] EDAC MC: Ver: 3.0.0 [ 0.669161] PCI: Using ACPI for IRQ routing [ 0.671979] NetLabel: Initializing [ 0.673014] NetLabel: domain hash size = 128 [ 0.674009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676092] NetLabel: unlabeled traffic allowed by default [ 0.678124] vgaarb: loaded [ 0.681355] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.683013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.690045] clocksource: Switched to clocksource kvm-clock [ 0.800759] VFS: Disk quotas dquot_6.6.0 [ 0.802411] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.805030] *** VALIDATE ramfs *** [ 0.806277] *** VALIDATE hugetlbfs *** [ 0.807851] pnp: PnP ACPI init [ 0.810083] pnp: PnP ACPI: found 6 devices [ 0.841873] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.845203] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.847184] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.849142] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.851367] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.853643] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.856545] NET: Registered protocol family 2 [ 0.858718] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.863299] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.866612] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.871652] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.875296] TCP: Hash tables configured (established 65536 bind 65536) [ 0.877893] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.880382] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.882555] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.885253] NET: Registered protocol family 1 [ 0.887785] RPC: Registered named UNIX socket transport module. [ 0.889656] RPC: Registered udp transport module. [ 0.891286] RPC: Registered tcp transport module. [ 0.892630] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.894553] NET: Registered protocol family 44 [ 0.895894] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.897740] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.899471] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.901462] PCI: CLS 0 bytes, default 64 [ 0.902875] Unpacking initramfs... [ 2.345826] debug: unmapping init [mem 0xffff983abcc54000-0xffff983abffbffff] [ 2.349915] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.352190] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.355063] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.860395] Initialise system trusted keyrings [ 2.862255] Key type blacklist registered [ 2.864483] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.873731] zbud: loaded [ 2.877034] *** VALIDATE nfs *** [ 2.878087] *** VALIDATE nfs4 *** [ 2.879638] pstore: using deflate compression [ 2.886932] Platform Keyring initialized [ 3.011798] NET: Registered protocol family 38 [ 3.014290] Key type asymmetric registered [ 3.015910] Asymmetric key parser 'x509' registered [ 3.018183] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.021540] io scheduler mq-deadline registered [ 3.023695] io scheduler kyber registered [ 3.025588] io scheduler bfq registered [ 3.027620] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.030696] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.033947] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.036852] ACPI: Power Button [PWRF] [ 3.042383] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.049552] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.065313] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.073541] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.094772] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.122945] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.153296] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.158745] Non-volatile memory driver v1.3 [ 3.160498] Linux agpgart interface v0.103 [ 3.199226] virtio_blk virtio1: [vda] 147232 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.202306] vda: detected capacity change from 0 to 75382784 [ 3.219853] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.222631] vdb: detected capacity change from 0 to 1073741824 [ 3.244492] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.247582] vdc: detected capacity change from 0 to 2621440000 [ 3.264817] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.267868] vdd: detected capacity change from 0 to 2621440000 [ 3.281799] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.284797] vde: detected capacity change from 0 to 4294967296 [ 3.302270] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.305107] vdf: detected capacity change from 0 to 4294967296 [ 3.313810] libphy: Fixed MDIO Bus: probed [ 3.323913] usbcore: registered new interface driver usbserial_generic [ 3.326544] usbserial: USB Serial support registered for generic [ 3.328693] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.332916] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.334578] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.337204] mousedev: PS/2 mouse device common for all mice [ 3.340037] rtc_cmos 00:05: RTC can wake from S4 [ 3.342614] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.343026] rtc_cmos 00:05: registered as rtc0 [ 3.347592] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.350388] intel_pstate: CPU model not supported [ 3.353538] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.356549] hid: raw HID events driver (C) Jiri Kosina [ 3.359846] usbcore: registered new interface driver usbhid [ 3.359938] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.362101] usbhid: USB HID core driver [ 3.366911] drop_monitor: Initializing network drop monitor service [ 3.369425] Initializing XFRM netlink socket [ 3.371406] NET: Registered protocol family 10 [ 3.373876] Segment Routing with IPv6 [ 3.375324] NET: Registered protocol family 17 [ 3.377537] mpls_gso: MPLS GSO support [ 3.384582] RAS: Correctable Errors collector initialized. [ 3.386784] AVX version of gcm_enc/dec engaged. [ 3.388532] AES CTR mode by8 optimization enabled [ 3.468924] sched_clock: Marking stable (3468902622, 0)->(4481120292, -1012217670) [ 3.472475] registered taskstats version 1 [ 3.474420] Loading compiled-in X.509 certificates [ 3.476344] zswap: loaded using pool lzo/zbud [ 3.502558] Key type big_key registered [ 3.517976] Key type encrypted registered [ 3.519870] ima: No TPM chip found, activating TPM-bypass! [ 3.522221] ima: Allocated hash algorithm: sha1 [ 3.524087] ima: No architecture policies found [ 3.526041] evm: Initialising EVM extended attributes: [ 3.528170] evm: security.selinux [ 3.529335] evm: security.ima [ 3.530199] evm: security.capability [ 3.531062] evm: HMAC attrs: 0x1 [ 3.534810] rtc_cmos 00:05: setting system clock to 2026-09-11 09:23:21 UTC (1789118601) [ 3.541712] debug: unmapping init [mem 0xffffffffa5403000-0xffffffffa55fffff] [ 3.545208] debug: unmapping init [mem 0xffffffffa4182000-0xffffffffa4458fff] [ 3.555115] Write protecting the kernel read-only data: 28672k [ 3.558640] debug: unmapping init [mem 0xffffffffa2803000-0xffffffffa29fffff] [ 3.561829] debug: unmapping init [mem 0xffffffffa3114000-0xffffffffa31fffff] [ 3.611700] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.621229] systemd[1]: Detected virtualization kvm. [ 3.623323] systemd[1]: Detected architecture x86-64. [ 3.625399] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.656443] systemd[1]: No hostname configured. [ 3.658475] systemd[1]: Set hostname to . [ 3.660725] random: systemd: uninitialized urandom read (16 bytes read) [ 3.663405] systemd[1]: Initializing machine ID from random generator. [ 3.705868] random: ln: uninitialized urandom read (6 bytes read) [ 3.820344] random: systemd: uninitialized urandom read (16 bytes read) [ 3.823069] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.828079] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.832619] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.521129] device-mapper: uevent: version 1.0.3 [ 4.523462] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.314327] virtio_net virtio0 ens2: renamed from eth0 [ 5.331888] random: fast init done [ 5.453550] scsi host0: ata_piix [ 5.497829] scsi host1: ata_piix [ 5.500376] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.502371] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.025684] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.039110] random: crng init done [ 10.040770] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.689427] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.982363] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.250352] SELinux: Disabled at runtime. [ 12.313888] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.322128] systemd[1]: Detected virtualization kvm. [ 12.324106] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.898526] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.902208] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.908746] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.913394] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.916531] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.923397] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.928806] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK [0[ 13.057750] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.402469] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.789206] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.803615] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.009633] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.020561] EDAC sbridge: Ver: 1.1.2 [ 15.730749] Key type dns_resolver registered [ 16.053476] NFS: Registering the id_resolver key type [ 16.055457] Key type id_resolver registered [ 16.057013] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg432-server login: [ 66.317768] hrtimer: interrupt took 2791519 ns [ 96.478721] libcfs: loading out-of-tree module taints kernel. [ 96.532880] Key type ._llcrypt registered [ 96.535375] Key type .llcrypt registered [ 96.682246] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 115.508597] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 116.688326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.709089] alg: No test for adler32 (adler32-zlib) [ 117.980133] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 118.621915] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 120.311830] Key type lgssc registered [ 121.983620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 135.723647] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 173.209287] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 185.497130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.535375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.818044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 186.858904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 186.962725] Lustre: lustre-MDT0000: new disk, initializing [ 187.091355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.123308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.305555] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.806613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.916671] Lustre: 6525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.980069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.984018] Lustre: Skipped 1 previous similar message [ 203.058408] Lustre: lustre-MDT0001: new disk, initializing [ 203.133730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.167544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.176243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.852764] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.701808] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.035108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.286841] Lustre: lustre-OST0000: new disk, initializing [ 220.298320] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.303431] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.357803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.820718] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.388289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 227.399470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 227.466779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 237.843836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.942928] Lustre: lustre-OST0001: new disk, initializing [ 237.946746] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.954681] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.012211] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.304900] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.316730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 246.329731] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 246.416664] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 254.373576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.670733] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.236946] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.174138] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing yml_node [ 274.782826] Lustre: DEBUG MARKER: Client: 2.17.57.140 [ 277.005472] Lustre: DEBUG MARKER: MDS: 2.17.57.140 [ 279.227672] Lustre: DEBUG MARKER: OSS: 2.17.57.140 [ 281.026648] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 05:27:57 EDT 2026 [ 294.887531] Lustre: DEBUG MARKER: excepting tests: 102 106 115 32newtarball [ 296.286386] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.800725] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.803133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 312.818786] Lustre: Skipped 1 previous similar message [ 317.792846] Lustre: server umount lustre-MDT0000 complete [ 321.707229] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118919 with bad export cookie 8773070943348977616 [ 321.716216] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.719123] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 323.040133] LustreError: 6534:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.040302] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.040854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.040858] Lustre: Skipped 3 previous similar messages [ 323.065820] LustreError: 6534:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 323.076490] Lustre: Skipped 3 previous similar messages [ 326.048374] LustreError: 6532:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.049301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.083188] Lustre: Skipped 2 previous similar messages [ 331.169524] LustreError: 10111:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.170648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.186721] LustreError: 10111:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 331.198178] Lustre: Skipped 1 previous similar message [ 336.351219] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 336.352876] LustreError: 6532:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.352963] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.383456] LustreError: 6532:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 336.629995] Lustre: server umount lustre-MDT0001 complete [ 339.876363] Lustre: server umount lustre-OST0000 complete [ 343.629867] Lustre: server umount lustre-OST0001 complete [ 350.550433] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 360.430790] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 391.507230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.816242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 391.842790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 391.903781] Lustre: lustre-MDT0000: new disk, initializing [ 391.991014] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.010427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 396.350553] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 406.572836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.733322] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 406.814517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 406.819925] Lustre: Skipped 1 previous similar message [ 406.900321] Lustre: lustre-MDT0001: new disk, initializing [ 407.032291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.110059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 407.119988] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 411.441135] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 421.073906] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 426.820947] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 433.157814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.452397] Lustre: lustre-OST0000: new disk, initializing [ 433.455995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 433.462217] Lustre: 17058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 433.542488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.138351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.154941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.248199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 439.041500] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 447.809270] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 452.702238] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 453.027872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 458.384243] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 458.598701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.892360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.899299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.932072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.950580] Lustre: Skipped 2 previous similar messages [ 467.241450] Lustre: server umount lustre-OST0000 complete [ 473.576564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 473.588249] LustreError: Skipped 1 previous similar message [ 473.595145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.614843] Lustre: Skipped 1 previous similar message [ 477.463246] Lustre: server umount lustre-MDT0000 complete [ 480.484232] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119078 with bad export cookie 8773070943348979149 [ 480.488342] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 480.495022] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 480.760945] Lustre: server umount lustre-MDT0001 complete [ 485.457986] Lustre: DEBUG MARKER: == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 05:31:22 (1789119082) [ 490.979336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.357927] LustreError: 19321:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.436123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.859094] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 501.734136] LustreError: 19322:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.759068] LustreError: 19322:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 503.046852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.169706] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 516.750918] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 523.518063] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 531.761113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 539.091465] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 547.504880] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 554.909721] Lustre: Modifying parameter general.osc.*.max_dirty_mb=452 in log params [ 564.199069] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.207633] Lustre: Skipped 1 previous similar message [ 564.216992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 564.220921] Lustre: Skipped 5 previous similar messages [ 567.352671] Lustre: server umount lustre-MDT0000 complete [ 569.317106] LustreError: 19317:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.335684] LustreError: 19317:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 571.010885] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119168 with bad export cookie 8773070943348980031 [ 571.019636] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 571.031390] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 571.484253] Lustre: server umount lustre-MDT0001 complete [ 585.347531] Lustre: server umount lustre-OST0000 complete [ 606.876261] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 616.712305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.257748] LustreError: 23703:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.293402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 617.299651] Lustre: Skipped 2 previous similar messages [ 622.642418] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 632.163127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.319574] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 642.936693] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 651.373980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 660.124354] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 670.740403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 671.016515] Lustre: lustre-OST0001: new disk, initializing [ 671.022761] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 671.037604] Lustre: 26022:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 672.357126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 672.377598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 672.501606] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 678.709199] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 689.308199] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 693.186903] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 708.606688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.613277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.637115] Lustre: Skipped 2 previous similar messages [ 708.655232] Lustre: Skipped 3 previous similar messages [ 712.643122] Lustre: server umount lustre-MDT0000 complete [ 713.698486] LustreError: 24844:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 713.733911] LustreError: 24844:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 717.220514] LustreError: 26021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119315 with bad export cookie 8773070943348980878 [ 717.237417] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 717.243575] LustreError: 26021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 717.856283] Lustre: server umount lustre-MDT0001 complete [ 732.544655] Lustre: server umount lustre-OST0000 complete [ 734.507349] Lustre: 3661:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789119316/real 1789119316] req@ffff983b1808da40 x1876026946989184/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789119332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 734.545481] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.564879] Lustre: Skipped 1 previous similar message [ 737.744320] Lustre: server umount lustre-OST0001 complete [ 759.777681] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 770.760199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.297427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.314955] Lustre: Skipped 3 previous similar messages [ 775.849911] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 784.544045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.654769] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 792.743653] Lustre: 30191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 799.492364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 806.278461] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 815.693762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 822.255256] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 830.589143] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 834.271796] Lustre: 32070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 841.186098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 841.190630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.199910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 846.643228] Lustre: server umount lustre-MDT0000 complete [ 847.841148] LustreError: 29043:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 847.877011] LustreError: 29043:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 849.973205] LustreError: 29025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119447 with bad export cookie 8773070943348982173 [ 849.982141] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 849.997419] LustreError: 29025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 850.312223] Lustre: server umount lustre-MDT0001 complete [ 864.527045] Lustre: server umount lustre-OST0000 complete [ 877.339398] Lustre: server umount lustre-OST0001 complete [ 885.096160] Lustre: DEBUG MARKER: == conf-sanity test 76b: verify params log setup correctly ========================================================== 05:38:01 (1789119481) [ 913.385899] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 923.726634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.174900] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 940.100504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.179955] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 949.758615] Lustre: 36023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 957.225285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 964.704199] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 974.665749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 981.850147] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 990.594982] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 994.420356] Lustre: 37901:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1000.935846] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.939182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.960701] Lustre: Skipped 5 previous similar messages [ 1000.984445] Lustre: Skipped 7 previous similar messages [ 1006.029184] Lustre: server umount lustre-MDT0000 complete [ 1009.950202] LustreError: 34855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119607 with bad export cookie 8773070943348983279 [ 1009.974738] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.611345] Lustre: server umount lustre-MDT0001 complete [ 1025.568538] Lustre: server umount lustre-OST0000 complete [ 1027.551187] Lustre: 3661:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789119609/real 1789119609] req@ffff983a041b9e00 x1876026947090560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789119625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1030.217894] Lustre: server umount lustre-OST0001 complete [ 1039.248553] Lustre: DEBUG MARKER: == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 05:40:35 (1789119635) [ 1057.277890] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1067.597323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.193833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.207631] Lustre: Skipped 7 previous similar messages [ 1073.831300] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1083.046691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.295571] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1091.643769] Lustre: 41193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1098.453062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.670461] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1108.991899] LustreError: 41549:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.017252] LustreError: 41549:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [ 1115.905725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.771190] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1132.089265] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1135.763736] Lustre: 43070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1136.736354] Lustre: Setting parameter general.mdd.*.changelog_mask=-CLOSE in log params [ 1136.744507] Lustre: Skipped 1 previous similar message [ 1142.249554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.276941] Lustre: Skipped 4 previous similar messages [ 1146.370896] Lustre: server umount lustre-MDT0000 complete [ 1150.254473] LustreError: 41195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119748 with bad export cookie 8773070943348984385 [ 1150.263700] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.273191] LustreError: 41195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1156.667720] Lustre: server umount lustre-MDT0001 complete [ 1166.362501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1176.036635] LustreError: 3660:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff983a059112c0 x1876026947142272/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1181.119694] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1186.034051] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1189.603970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.875144] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1195.495563] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1195.506821] Lustre: Skipped 1 previous similar message [ 1202.720779] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1204.449055] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1211.724172] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1213.659793] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1220.583885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1225.985639] Lustre: server umount lustre-MDT0000 complete [ 1230.081151] LustreError: 40026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119828 with bad export cookie 8773070943348985512 [ 1230.086558] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.096787] LustreError: 40026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1230.865249] Lustre: server umount lustre-MDT0001 complete [ 1245.817843] Lustre: server umount lustre-OST0000 complete [ 1247.714155] Lustre: 3663:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789119829/real 1789119829] req@ffff983b1808d680 x1876026947178368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789119845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1249.800562] Lustre: server umount lustre-OST0001 complete [ 1259.103403] Lustre: DEBUG MARKER: == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 05:44:15 (1789119855) [ 1272.934319] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1283.951448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.995967] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1300.318451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.992992] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1308.094486] Lustre: 48831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1314.989507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1321.221176] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1329.940576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1336.912711] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1344.596257] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1348.816833] Lustre: Setting parameter general.llite.*.xattr_cache=0 in log params [ 1361.898328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1361.981089] Lustre: server umount lustre-MDT0000 complete [ 1366.473382] LustreError: 47660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119964 with bad export cookie 8773070943348986310 [ 1366.484085] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1366.487736] LustreError: 47660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1366.839733] Lustre: server umount lustre-MDT0001 complete [ 1380.623876] Lustre: server umount lustre-OST0000 complete [ 1396.129855] Lustre: server umount lustre-OST0001 complete [ 1409.050631] Lustre: DEBUG MARKER: == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 05:46:45 (1789120005) [ 1412.092481] Lustre: DEBUG MARKER: SKIP: conf-sanity test_77 mixed loopback and real device not working [ 1414.518783] Lustre: DEBUG MARKER: == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 05:46:50 (1789120010) [ 1427.916718] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 1438.569621] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1463.596750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.015199] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1464.092882] Lustre: lustre-MDT0000: new disk, initializing [ 1464.199512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1468.259819] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1479.319342] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1487.883314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1488.084242] Lustre: lustre-OST0000: new disk, initializing [ 1488.087900] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1488.092351] Lustre: Skipped 1 previous similar message [ 1488.096625] Lustre: 54912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1490.218753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1490.237177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1490.269710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 1496.303972] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1506.941872] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1510.175373] Lustre: DEBUG MARKER: create test files [ 1546.722088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1546.746588] Lustre: Skipped 12 previous similar messages [ 1546.761581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1546.774975] Lustre: Skipped 15 previous similar messages [ 1551.157398] Lustre: server umount lustre-OST0000 complete [ 1555.258482] Lustre: server umount lustre-MDT0000 complete [ 1567.256497] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1570.633402] Key type lgssc unregistered [ 1570.988670] LNet: 56549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.017657] LNetError: 56549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.035475] LNet: Removed LNI 192.168.204.132@tcp [ 1572.097187] Key type .llcrypt unregistered [ 1572.101082] Key type ._llcrypt unregistered [ 1601.699352] Key type ._llcrypt registered [ 1601.702768] Key type .llcrypt registered [ 1601.830433] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1602.949802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1603.160061] alg: No test for adler32 (adler32-zlib) [ 1604.325492] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 1604.599059] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1606.297083] Key type lgssc registered [ 1607.444443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.047871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1615.084055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.571909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.177950] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1629.857974] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1638.011663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1638.263943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1641.349539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:102 to 0x240000400:129) [ 1641.501622] Lustre: 59285:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 511 < left 544, rollback = 9 [ 1641.525156] Lustre: 59285:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/4, destroy: 0/0/0 [ 1641.547793] Lustre: 59285:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1641.560821] Lustre: 59285:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1641.568903] Lustre: 59285:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1641.579249] Lustre: 59285:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1645.479821] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1654.296643] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1659.786239] Lustre: DEBUG MARKER: check files after expanding the MDT and OST filesystems [ 1665.903346] Lustre: DEBUG MARKER: create more files after expanding the MDT and OST filesystems [ 1675.550816] Lustre: server umount lustre-OST0000 complete [ 1690.475847] Lustre: server umount lustre-MDT0000 complete [ 1702.847541] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1705.540355] Key type lgssc unregistered [ 1705.814345] LNet: 60900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.821995] LNetError: 60900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.840264] LNet: Removed LNI 192.168.204.132@tcp [ 1706.836327] Key type .llcrypt unregistered [ 1706.845293] Key type ._llcrypt unregistered [ 1738.410951] Key type ._llcrypt registered [ 1738.413129] Key type .llcrypt registered [ 1738.556391] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1739.752620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1739.791144] alg: No test for adler32 (adler32-zlib) [ 1740.830763] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 1741.040406] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1742.719156] Key type lgssc registered [ 1743.999867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1752.465262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1752.499237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.054299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1758.630743] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1766.670799] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1773.539154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.885980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.989941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:140 to 0x240000400:161) [ 1779.062311] Lustre: 63734:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 8: before 509 < left 544, rollback = 9 [ 1779.067812] Lustre: 63734:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 32/128/6, destroy: 0/0/0 [ 1779.071028] Lustre: 63734:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 0/0/0 [ 1779.075406] Lustre: 63734:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/18/1, punch: 0/0/0, quota 1/3/0 [ 1779.079816] Lustre: 63734:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 32/544/0, delete: 0/0/0 [ 1779.088951] Lustre: 63734:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1779.986987] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1788.751763] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1795.050559] Lustre: DEBUG MARKER: check files after shrinking the MDT and OST filesystems [ 1809.888260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1809.893958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.916911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1813.213587] Lustre: server umount lustre-OST0000 complete [ 1818.226830] Lustre: server umount lustre-MDT0000 complete [ 1833.328392] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1837.264280] Key type lgssc unregistered [ 1837.657352] LNet: 65294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.675648] LNetError: 65294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1838.695748] LNet: Removed LNI 192.168.204.132@tcp [ 1840.219045] Key type .llcrypt unregistered [ 1840.226593] Key type ._llcrypt unregistered [ 1862.521961] Key type ._llcrypt registered [ 1862.526852] Key type .llcrypt registered [ 1862.736734] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 1880.087381] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1881.428533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1881.552476] alg: No test for adler32 (adler32-zlib) [ 1882.695498] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 1882.927471] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1884.679394] Key type lgssc registered [ 1885.703853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1925.592383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1925.635532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.879906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1926.917656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1927.057323] Lustre: lustre-MDT0000: new disk, initializing [ 1927.151159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1927.187524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1932.141261] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1945.055975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.257187] Lustre: 68447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1945.342822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1945.347111] Lustre: Skipped 1 previous similar message [ 1945.428873] Lustre: lustre-MDT0001: new disk, initializing [ 1945.551493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1945.611414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1945.619619] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1951.192430] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1961.658643] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1969.052913] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1978.351359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1978.936266] Lustre: lustre-OST0000: new disk, initializing [ 1978.940316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1978.948256] Lustre: 70419:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1979.103196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.621943] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1986.677756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1986.701769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1986.871606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1999.804467] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2006.337627] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2006.728928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2013.250755] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2013.714355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2017.796415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2017.800758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.812500] LustreError: Skipped 1 previous similar message [ 2017.814096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2017.891303] Lustre: Skipped 1 previous similar message [ 2022.696318] Lustre: server umount lustre-OST0000 complete [ 2027.491695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2027.501571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.513756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2027.519207] Lustre: Skipped 1 previous similar message [ 2033.123484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.139071] Lustre: Skipped 2 previous similar messages [ 2033.981914] Lustre: server umount lustre-MDT0000 complete [ 2038.863776] LustreError: 70418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120636 with bad export cookie 9501593589117271509 [ 2038.879021] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2038.885182] LustreError: 70418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2039.490677] Lustre: server umount lustre-MDT0001 complete [ 2050.409458] Lustre: DEBUG MARKER: == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 05:57:26 (1789120646) [ 2063.974646] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2112.733452] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2125.138593] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2168.521747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.901967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2168.945807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2169.064726] Lustre: lustre-MDT0000: new disk, initializing [ 2169.199215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.239088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2174.664250] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2187.124967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2187.304691] Lustre: 76440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2187.369073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2187.372419] Lustre: Skipped 1 previous similar message [ 2187.548430] Lustre: lustre-MDT0001: new disk, initializing [ 2187.716920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.801471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2187.817270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2193.932896] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2205.215385] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2213.468809] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2222.549373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2222.837987] Lustre: lustre-OST0000: new disk, initializing [ 2222.841793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2222.848926] Lustre: 78409:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2222.936819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.625497] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2232.408734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2232.426283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2232.531120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2242.220878] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2248.468823] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2248.699934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2254.292716] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2254.499264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2258.421874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.428556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2258.440231] Lustre: Skipped 2 previous similar messages [ 2258.456406] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2263.532485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2263.546738] Lustre: Skipped 2 previous similar messages [ 2271.199225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2271.382483] Lustre: server umount lustre-OST0000 complete [ 2278.882822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.887068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.896532] Lustre: Skipped 1 previous similar message [ 2278.908229] Lustre: Skipped 2 previous similar messages [ 2282.520841] Lustre: server umount lustre-MDT0000 complete [ 2284.004831] LustreError: 76452:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.024438] LustreError: 76452:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2287.682282] LustreError: 76431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120885 with bad export cookie 9501593589117272391 [ 2287.686273] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.703472] LustreError: 76431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2288.444732] Lustre: server umount lustre-MDT0001 complete [ 2300.748100] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ======== 06:01:36 (1789120896) [ 2308.550370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.976442] LustreError: 80839:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2309.068666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2314.210490] LustreError: 80840:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2314.228544] LustreError: 80840:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2314.304409] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2318.315900] LustreError: 80839:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.425132] LustreError: 80840:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.758624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.837695] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2342.530909] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2349.464723] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2357.709046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2358.196761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.208650] Lustre: Skipped 1 previous similar message [ 2366.070232] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2374.643666] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2378.900531] Lustre: 83511:0:(genops.c:1774:obd_export_evict_by_uuid()) MGS: evicting 931baf57-4b28-455e-9de4-241123df0c65 at adminstrative request [ 2383.852916] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.886278] Lustre: *** cfs_fail_loc=906, val=2147483648*** [ 2393.078494] Lustre: MGS: Client 931baf57-4b28-455e-9de4-241123df0c65 (at 0@lo) reconnecting [ 2393.092859] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x83dc71071776c4b9 to 0x83dc71071776c759 [ 2393.110475] Lustre: MGC192.168.204.132@tcp: Connection restored to 0@lo (at 0@lo) [ 2418.153534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2418.564554] Lustre: lustre-OST0001: new disk, initializing [ 2418.576390] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2418.584131] Lustre: 83908:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2424.948940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2424.966614] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2425.174533] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2426.981857] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2439.195187] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2445.796464] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2445.798840] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.812985] LustreError: Skipped 1 previous similar message [ 2445.814228] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2445.814233] Lustre: Skipped 1 previous similar message [ 2445.862605] Lustre: Skipped 1 previous similar message [ 2448.805784] Lustre: server umount lustre-OST0001 complete [ 2450.918898] LustreError: 82662:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.944698] LustreError: 82662:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2453.479419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2455.520601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.549430] Lustre: Skipped 1 previous similar message [ 2467.295861] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2467.586689] Lustre: server umount lustre-OST0000 complete [ 2474.982884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2475.004207] LustreError: Skipped 1 previous similar message [ 2475.015225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.054805] Lustre: server umount lustre-MDT0000 complete [ 2481.633627] LustreError: 80839:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.648976] LustreError: 80839:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 2484.329989] LustreError: 83907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121082 with bad export cookie 9501593589117273945 [ 2484.343245] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2484.347139] LustreError: 83907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2484.960112] Lustre: server umount lustre-MDT0001 complete [ 2494.845223] Lustre: DEBUG MARKER: == conf-sanity test 81: sparse OST indexing ============== 06:04:50 (1789121090) [ 2496.917439] Lustre: DEBUG MARKER: SKIP: conf-sanity test_81 needs >= 3 OSTs [ 2498.900384] Lustre: DEBUG MARKER: == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 06:04:55 (1789121095) [ 2500.498380] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82a needs >= 3 OSTs [ 2502.613459] Lustre: DEBUG MARKER: == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 06:04:58 (1789121098) [ 2504.428624] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82b needs >= 4 OSTs [ 2506.315175] Lustre: DEBUG MARKER: == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 06:05:02 (1789121102) [ 2508.454500] Lustre: DEBUG MARKER: SKIP: conf-sanity test_82c needs >= 4 OSTs [ 2510.570616] Lustre: DEBUG MARKER: == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 06:05:06 (1789121106) [ 2512.603308] Lustre: DEBUG MARKER: mount the OST /dev/mapper/ost1_flakey as a ldiskfs filesystem [ 2521.738804] loop: module loaded [ 2521.773138] loop0: detected capacity change from 0 to 4294967296 [ 2521.882303] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 2524.358866] Lustre: DEBUG MARKER: run llverfs in partial mode on the OST ldiskfs /mnt/lustre-ost1 [ 2530.067190] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing run_llverfs /mnt/lustre-ost1 -vpl no [ 2534.912301] Lustre: DEBUG MARKER: unmount the OST /dev/mapper/ost1_flakey [ 2565.285870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2565.348475] LustreError: 88035:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-28) [ 2565.362921] LustreError: 88035:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -28 [ 2565.378881] LustreError: 88035:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -28 [ 2565.398660] LustreError: 88035:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -28 [ 2587.257911] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2600.254995] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2643.125511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.307786] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2643.333612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2643.462804] Lustre: lustre-MDT0000: new disk, initializing [ 2643.549390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2643.553502] Lustre: Skipped 1 previous similar message [ 2643.573223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2648.438764] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2659.397276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.512460] Lustre: 90904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2659.828380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2664.553737] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2674.465810] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2680.718831] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2688.781457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2689.131159] Lustre: 92872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2695.394853] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2698.857340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2699.013480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2706.349432] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2712.385173] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2712.660309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2718.456844] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2718.822393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2724.837831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2724.838063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2724.841275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2724.841287] Lustre: Skipped 12 previous similar messages [ 2724.874087] Lustre: Skipped 2 previous similar messages [ 2735.072878] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2735.285227] Lustre: server umount lustre-OST0000 complete [ 2746.326294] Lustre: server umount lustre-MDT0000 complete [ 2750.373847] LustreError: 91700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121348 with bad export cookie 9501593589117274561 [ 2750.380297] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2750.394579] LustreError: 91700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2750.751111] Lustre: server umount lustre-MDT0001 complete [ 2761.331742] Lustre: DEBUG MARKER: == conf-sanity test 84: check recovery_hard_time ========= 06:09:17 (1789121357) [ 2768.746632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.297787] LustreError: 95287:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.813981] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2785.290890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2785.809827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-60 [ 2785.821752] Lustre: Skipped 3 previous similar messages [ 2792.204285] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2802.619754] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2810.392404] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2818.073217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2825.665717] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2833.870631] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2839.403365] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2840.745559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2846.359189] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2846.672228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2855.109818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2855.251382] Lustre: lustre-OST0001: new disk, initializing [ 2855.253227] Lustre: Skipped 2 previous similar messages [ 2855.256652] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2855.260200] Lustre: Skipped 3 previous similar messages [ 2855.263958] Lustre: 98666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2862.374889] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2862.735942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2862.747971] Lustre: Skipped 2 previous similar messages [ 2862.752429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2862.807785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2872.962951] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2878.819442] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2879.122151] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2884.443500] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2884.736571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2906.797796] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2934.380282] Lustre: Failing over lustre-MDT0000 [ 2934.763433] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.774485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2934.786231] Lustre: Skipped 4 previous similar messages [ 2934.807231] Lustre: Skipped 11 previous similar messages [ 2934.915510] Lustre: server umount lustre-MDT0000 complete [ 2938.149728] LustreError: 100066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.167634] LustreError: 100066:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2939.362672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2939.373827] LustreError: Skipped 1 previous similar message [ 2955.238236] Lustre: 66673:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789121537/real 1789121537] req@00000000fb0b01b1 x1876028796733312/t0(0) o400->MGC192.168.204.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789121553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2955.272830] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2960.678218] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2960.680711] LDISKFS-fs (dm-0): recovery complete [ 2960.691336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.476303] LustreError: 66669:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004537521b x1876028796741632/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2965.809515] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2968.853226] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2968.853833] Lustre: 95283:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2970.215923] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2971.119309] Lustre: 95288:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [ 2971.133413] Lustre: 95288:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 1 previous similar message [ 2971.153983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2971.253094] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2971.575105] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2971.914684] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2971.931279] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 1 previous similar message [ 2972.247117] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2972.255960] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 1 previous similar message [ 2972.271845] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2972.293835] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 7 previous similar messages [ 2972.944480] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2972.958360] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 2 previous similar messages [ 2973.287849] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2973.300129] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 2 previous similar messages [ 2974.280577] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2974.288994] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 5 previous similar messages [ 2975.270321] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2975.282450] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 6 previous similar messages [ 2975.601665] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2975.613504] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 6 previous similar messages [ 2978.489753] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2978.501765] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 12 previous similar messages [ 2979.449323] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2979.458804] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 12 previous similar messages [ 2979.791220] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2979.798830] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 12 previous similar messages [ 2986.806994] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 2986.823960] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 25 previous similar messages [ 2987.463865] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 2987.472254] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 24 previous similar messages [ 2987.802178] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 2987.816695] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 24 previous similar messages [ 3002.835065] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [ 3002.866901] Lustre: 101515:0:(ldlm_lib.c:2148:extend_recovery_timer()) Skipped 49 previous similar messages [ 3003.537682] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 sleeping for 300ms [ 3003.547840] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 49 previous similar messages [ 3003.895201] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) cfs_fail_timeout id 709 awake [ 3003.898842] LustreError: 101515:0:(ldlm_lib.c:2761:replay_request_or_update()) Skipped 49 previous similar messages [ 3028.199175] Lustre: 101515:0:(ldlm_lib.c:2457:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 3028.212641] Lustre: 101515:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3028.221260] Lustre: 101515:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 918b553d-96c8-4ccb-9dab-0446e15cac19@192.168.204.32@tcp [ 3028.230284] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 3028.235897] Lustre: 101515:0:(ldlm_lib.c:1977:abort_req_replay_queue()) @@@ aborted: req@00000000ae48ea59 x1876028778681216/t0(8589934774) o35->918b553d-96c8-4ccb-9dab-0446e15cac19@192.168.204.32@tcp:401/0 lens 392/0 e 0 to 0 dl 1789121636 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:0 [ 3028.258545] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3028.276795] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3028.295161] Lustre: lustre-MDT0000: Denying connection for new client 918b553d-96c8-4ccb-9dab-0446e15cac19 (at 192.168.204.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 1 evicted) to recover in 0:00 [ 3028.316752] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3028.343809] Lustre: lustre-MDT0000: Recovery over after 1:00, of 3 clients 2 recovered and 1 was evicted. [ 3028.344448] Lustre: Skipped 2 previous similar messages [ 3028.414953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:47 to 0x2c0000401:545) [ 3028.417948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:78 to 0x280000401:577) [ 3062.239357] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3062.600652] Lustre: server umount lustre-OST0000 complete [ 3067.877120] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3067.889081] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.902833] Lustre: Skipped 3 previous similar messages [ 3067.918148] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3067.926500] Lustre: Skipped 6 previous similar messages [ 3068.391332] LustreError: 97111:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.416931] LustreError: 97111:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 41 previous similar messages [ 3073.959760] Lustre: server umount lustre-OST0001 complete [ 3085.292501] Lustre: server umount lustre-MDT0000 complete [ 3090.053117] LustreError: 95270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121688 with bad export cookie 9501593589117325780 [ 3090.065070] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3090.076548] LustreError: 95270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3090.825702] Lustre: server umount lustre-MDT0001 complete [ 3103.402352] Lustre: DEBUG MARKER: == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 06:14:59 (1789121699) [ 3113.607882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.620047] Lustre: *** cfs_fail_loc=197, val=0*** [ 3129.311659] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3134.431731] LustreError: 103496:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 3139.666590] Lustre: *** cfs_fail_loc=197, val=0*** [ 3139.671659] Lustre: Skipped 65 previous similar messages [ 3160.223334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.233435] Lustre: Skipped 3 previous similar messages [ 3166.236189] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3176.255398] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3178.959844] Lustre: server umount lustre-OST0000 complete [ 3190.872426] Lustre: DEBUG MARKER: == conf-sanity test 86: Replacing mkfs.lustre -G option == 06:16:26 (1789121786) [ 3214.680737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.585834] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3233.897845] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3239.064584] LustreError: 105146:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3239.080659] LustreError: 105146:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3239.102549] LustreError: 105146:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 3239.565385] Lustre: server umount lustre-MDT0000 complete [ 3282.210772] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 3294.127941] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3335.134914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.406864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3335.430376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3335.505549] Lustre: lustre-MDT0000: new disk, initializing [ 3335.570983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3340.383903] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3351.986905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3352.106110] Lustre: 109324:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3352.511138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3358.400764] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3369.319355] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3376.313503] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3383.916254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3384.313108] Lustre: 111297:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3391.107393] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3392.694958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3400.907820] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3406.108826] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3406.432744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3410.752971] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3411.041918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3413.480316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3413.498415] LustreError: Skipped 1 previous similar message [ 3413.502715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.536374] Lustre: Skipped 3 previous similar messages [ 3413.562583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3413.574035] Lustre: Skipped 7 previous similar messages [ 3419.470927] Lustre: server umount lustre-OST0000 complete [ 3430.963031] Lustre: server umount lustre-MDT0000 complete [ 3433.953562] LustreError: 111724:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.990155] LustreError: 111724:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [ 3435.896122] LustreError: 109316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122033 with bad export cookie 9501593589117336413 [ 3435.900613] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3435.906182] LustreError: 109316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3436.360619] Lustre: server umount lustre-MDT0001 complete [ 3446.169268] Lustre: DEBUG MARKER: == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 06:20:42 (1789122042) [ 3456.626479] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3459.755459] Key type lgssc unregistered [ 3460.053701] LNet: 113785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3460.062959] LNetError: 113785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3461.100266] LNet: Removed LNI 192.168.204.132@tcp [ 3462.280068] Key type .llcrypt unregistered [ 3462.283321] Key type ._llcrypt unregistered [ 3482.767160] Key type ._llcrypt registered [ 3482.772260] Key type .llcrypt registered [ 3482.872829] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 3499.739774] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3500.883209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3501.067385] alg: No test for adler32 (adler32-zlib) [ 3502.146297] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 3502.365477] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3504.055206] Key type lgssc registered [ 3505.588776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3560.168996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3560.214487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3561.577220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3561.761516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3561.935799] Lustre: lustre-MDT0000: new disk, initializing [ 3562.098218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.146763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3567.212889] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3579.284387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3579.585602] Lustre: lustre-OST0000: new disk, initializing [ 3579.604659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3579.612696] Lustre: Skipped 1 previous similar message [ 3579.617483] Lustre: 117962:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3579.703237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3585.105614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3585.111531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3585.146780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3586.866736] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3600.371125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3600.646912] Lustre: lustre-OST0001: new disk, initializing [ 3600.649486] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3600.655690] Lustre: 118826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3600.716942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3609.864737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:ost [ 3609.885891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:ost] [ 3609.956198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x280000400 [ 3610.071664] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3620.323721] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.338262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.363509] Lustre: Skipped 1 previous similar message [ 3620.396535] Lustre: Skipped 1 previous similar message [ 3625.444543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3625.454073] Lustre: Skipped 1 previous similar message [ 3626.264148] Lustre: server umount lustre-MDT0000 complete [ 3634.872610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3643.437733] LustreError: 117149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122241 with bad export cookie 4060107370706685730 [ 3643.467598] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3643.478789] LustreError: 117149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3643.785557] Lustre: server umount lustre-OST0000 complete [ 3649.564233] Lustre: server umount lustre-OST0001 complete [ 3664.299099] Lustre: DEBUG MARKER: == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 06:24:19 (1789122259) [ 3696.507406] Lustre: DEBUG MARKER: == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 06:24:52 (1789122292) [ 3710.640632] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3737.912963] Lustre: DEBUG MARKER: == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 06:25:34 (1789122334) [ 3782.547301] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 3795.412425] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3841.788318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.148254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3842.213548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3842.327927] Lustre: lustre-MDT0000: new disk, initializing [ 3842.425075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3842.453304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3848.214835] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3863.025577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.287591] Lustre: 126610:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3863.384564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3863.394693] Lustre: Skipped 1 previous similar message [ 3863.559587] Lustre: lustre-MDT0001: new disk, initializing [ 3863.651109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3863.694537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3863.717848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3868.953666] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3879.567811] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3886.830441] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3894.123526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3894.418535] Lustre: lustre-OST0000: new disk, initializing [ 3894.426476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3894.437949] Lustre: 128579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3894.574307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.496069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3900.507163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3900.624754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3901.327494] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3910.899258] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3915.724445] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3916.035156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3921.146219] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3921.526418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3926.502348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3926.502858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.516670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3926.523239] Lustre: Skipped 1 previous similar message [ 3931.626358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3931.633928] Lustre: Skipped 2 previous similar messages [ 3936.739410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3936.753994] Lustre: Skipped 1 previous similar message [ 3938.271418] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3938.696923] Lustre: server umount lustre-OST0000 complete [ 3945.459502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3945.470336] LustreError: Skipped 1 previous similar message [ 3945.475628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.507414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3950.121323] Lustre: server umount lustre-MDT0000 complete [ 3954.908455] LustreError: 128578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122552 with bad export cookie 4060107370706686955 [ 3954.910604] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3954.918369] LustreError: 128578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3955.308609] Lustre: server umount lustre-MDT0001 complete [ 3966.287580] Lustre: DEBUG MARKER: == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 06:29:22 (1789122562) [ 3974.307213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.014409] LustreError: 130992:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3975.043425] LustreError: 130992:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3975.140955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.265763] LustreError: 130993:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.396623] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3985.377591] LustreError: 130992:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3990.314680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.509610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3990.519812] Lustre: Skipped 2 previous similar messages [ 3995.469554] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4004.526734] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4011.365863] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4019.312660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.689323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.695218] Lustre: Skipped 1 previous similar message [ 4027.318156] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4036.112367] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4041.898808] Lustre: *** cfs_fail_loc=159, val=0*** [ 4057.440583] Lustre: lustre-MDT0000: Client b396b9c2-f648-4d71-bfe8-9c64e5f1b8e3 (at 192.168.204.32@tcp) reconnecting [ 4059.079434] Lustre: *** cfs_fail_loc=159, val=0*** [ 4059.082751] Lustre: Skipped 5 previous similar messages [ 4074.280315] Lustre: lustre-MDT0000: Client b396b9c2-f648-4d71-bfe8-9c64e5f1b8e3 (at 192.168.204.32@tcp) reconnecting [ 4096.488896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.491865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4096.506791] Lustre: Skipped 1 previous similar message [ 4096.510607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4097.460741] Lustre: server umount lustre-OST0000 complete [ 4103.137268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4103.146494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4103.167493] Lustre: Skipped 1 previous similar message [ 4110.925572] Lustre: server umount lustre-MDT0000 complete [ 4111.848595] LustreError: 130988:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.866643] LustreError: 130988:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4115.283455] LustreError: 130974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122713 with bad export cookie 4060107370706687837 [ 4115.295230] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4115.313248] LustreError: 130974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4115.705960] Lustre: server umount lustre-MDT0001 complete [ 4127.283553] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4130.709706] Key type lgssc unregistered [ 4131.369236] LNet: 134775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4131.374687] LNetError: 134775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4131.392699] LNet: Removed LNI 192.168.204.132@tcp [ 4132.372660] Key type .llcrypt unregistered [ 4132.375672] Key type ._llcrypt unregistered [ 4168.157702] Key type ._llcrypt registered [ 4168.162619] Key type .llcrypt registered [ 4168.376642] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4169.638415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4169.882793] alg: No test for adler32 (adler32-zlib) [ 4170.994645] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 4171.425699] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4173.175175] Key type lgssc registered [ 4174.411937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4183.570897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4183.604480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4185.092849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4189.154699] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4195.304855] LustreError: 136205:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4195.322306] LustreError: 136205:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4198.144417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4198.587912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4203.836977] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4214.133237] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4221.259484] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4230.695596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4231.105197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.923404] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4250.743627] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4255.676231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4267.481141] Lustre: *** cfs_fail_loc=159, val=0*** [ 4283.692122] Lustre: lustre-MDT0000: Client c597d86c-4b61-40c9-be2f-b8a333868395 (at 192.168.204.32@tcp) reconnecting [ 4287.482803] Lustre: *** cfs_fail_loc=159, val=0*** [ 4303.698211] Lustre: lustre-MDT0001: Client c597d86c-4b61-40c9-be2f-b8a333868395 (at 192.168.204.32@tcp) reconnecting [ 4305.652544] Lustre: *** cfs_fail_loc=159, val=0*** [ 4305.654540] Lustre: Skipped 3 previous similar messages [ 4322.136422] Lustre: lustre-MDT0001: Client c597d86c-4b61-40c9-be2f-b8a333868395 (at 192.168.204.32@tcp) reconnecting [ 4337.956857] Lustre: 139292:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4340.774132] Lustre: *** cfs_fail_loc=159, val=0*** [ 4340.787521] Lustre: Skipped 4 previous similar messages [ 4356.445685] Lustre: lustre-MDT0000: Client b7af1dae-2598-4dca-a263-3e43447fc262 (at 192.168.204.32@tcp) reconnecting [ 4358.513159] Lustre: *** cfs_fail_loc=159, val=0*** [ 4358.527822] Lustre: Skipped 17 previous similar messages [ 4374.847541] Lustre: lustre-MDT0000: Client b7af1dae-2598-4dca-a263-3e43447fc262 (at 192.168.204.32@tcp) reconnecting [ 4376.573152] Lustre: 139579:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4383.210593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 4383.216794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.219662] LustreError: Skipped 1 previous similar message [ 4383.255871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4383.713430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4383.728698] LustreError: Skipped 4 previous similar messages [ 4383.735564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.782417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4388.838650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4388.842085] Lustre: Skipped 1 previous similar message [ 4389.472399] Lustre: server umount lustre-OST0000 complete [ 4398.055955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4398.073406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.101198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4402.301189] Lustre: server umount lustre-MDT0000 complete [ 4404.195728] LustreError: 136206:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4404.241601] LustreError: 136206:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4407.012447] LustreError: 139170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123004 with bad export cookie 4115251076737913717 [ 4407.019502] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4407.027312] LustreError: 139170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4407.525561] Lustre: server umount lustre-MDT0001 complete [ 4421.531313] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4425.236315] Key type lgssc unregistered [ 4425.674713] LNet: 140637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4425.682741] LNetError: 140637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4425.711727] LNet: Removed LNI 192.168.204.132@tcp [ 4426.646573] Key type .llcrypt unregistered [ 4426.648691] Key type ._llcrypt unregistered [ 4457.264855] Key type ._llcrypt registered [ 4457.266994] Key type .llcrypt registered [ 4457.346918] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4458.511832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4458.728651] alg: No test for adler32 (adler32-zlib) [ 4459.934572] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 4460.334981] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4462.039361] Key type lgssc registered [ 4463.577901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4472.025747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4472.054215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4473.573545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.042431] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4483.045592] LustreError: 142053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.077330] LustreError: 142053:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4488.161366] LustreError: 142054:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.889654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.402814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4496.006368] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4505.832855] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4513.053807] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4522.624878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4523.237878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4530.865405] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4535.548811] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:97) [ 4540.251976] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4543.725455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:129) [ 4550.924596] Lustre: 144648:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4555.113651] Lustre: 144697:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4556.758575] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 in log params [ 4557.680614] Lustre: Setting parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=8 in log params [ 4562.395583] Lustre: 144844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre/mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight=8 (mode = 0) failed: rc = -17 [ 4563.385718] Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_mod_rpcs_in_flight=7 in log params [ 4564.532776] Lustre: 144952:0:(mdt_handler.c:64:mdt_max_mod_rpcs_per_client_set()) max_mod_rpcs_per_client is deprecated, set mdt.*.max_mod_rpcs_in_flight parameter instead [ 4571.616771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4571.624323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4571.653482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4574.690832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4574.699489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4574.699576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4574.725397] Lustre: Skipped 1 previous similar message [ 4576.665089] Lustre: server umount lustre-OST0000 complete [ 4582.382194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.392554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.436779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4584.928482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4584.929562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4584.951296] Lustre: Skipped 1 previous similar message [ 4590.050129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4590.064096] Lustre: Skipped 1 previous similar message [ 4595.167172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4595.416380] Lustre: server umount lustre-MDT0000 complete [ 4599.266734] LustreError: 142050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.283161] LustreError: 142050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4599.724811] LustreError: 144746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123197 with bad export cookie 6164279604025767158 [ 4599.739051] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.756881] LustreError: 144746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4600.322808] Lustre: server umount lustre-MDT0001 complete [ 4614.425963] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4617.464599] Key type lgssc unregistered [ 4617.924733] LNet: 146009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4617.937186] LNetError: 146009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4617.951655] LNet: Removed LNI 192.168.204.132@tcp [ 4619.082375] Key type .llcrypt unregistered [ 4619.087640] Key type ._llcrypt unregistered [ 4648.232540] Key type ._llcrypt registered [ 4648.235308] Key type .llcrypt registered [ 4648.353687] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4649.532614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4649.591310] alg: No test for adler32 (adler32-zlib) [ 4650.755056] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 4651.136957] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4652.911194] Key type lgssc registered [ 4654.248117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4665.108535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4665.136905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.037233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.584391] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4676.579217] LustreError: 147428:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.629978] LustreError: 147428:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4681.711500] LustreError: 147429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.462162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.278878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4692.593620] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4701.711543] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4708.741729] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4716.696791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4717.084582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4724.331876] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4731.517799] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:129) [ 4732.691850] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4736.623536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:85 to 0x280000401:161) [ 4745.191297] Lustre: *** cfs_fail_loc=159, val=0*** [ 4745.202676] Lustre: Skipped 1 previous similar message [ 4761.439794] Lustre: lustre-MDT0000: Client 1507c263-1b45-4b77-9271-f4c4201a5e72 (at 192.168.204.32@tcp) reconnecting [ 4767.219703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4767.232426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.259035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4769.357969] Lustre: server umount lustre-OST0000 complete [ 4777.971305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4777.983646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4778.001643] Lustre: Skipped 2 previous similar messages [ 4778.024383] Lustre: Skipped 2 previous similar messages [ 4780.574528] Lustre: server umount lustre-MDT0000 complete [ 4783.074388] LustreError: 149736:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4783.091430] LustreError: 149736:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4784.216222] LustreError: 147408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123382 with bad export cookie 13579599147920481152 [ 4784.223649] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4784.227690] LustreError: 147408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4784.766104] Lustre: server umount lustre-MDT0001 complete [ 4798.929775] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4802.276361] Key type lgssc unregistered [ 4802.606112] LNet: 151119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4802.623557] LNetError: 151119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4802.644856] LNet: Removed LNI 192.168.204.132@tcp [ 4803.317563] Key type .llcrypt unregistered [ 4803.320207] Key type ._llcrypt unregistered [ 4830.596711] Key type ._llcrypt registered [ 4830.600996] Key type .llcrypt registered [ 4830.754583] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4831.950977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4832.013863] alg: No test for adler32 (adler32-zlib) [ 4833.197298] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 4833.496474] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4835.175242] Key type lgssc registered [ 4836.373778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4845.670713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4845.688742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4847.300857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.217381] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4857.825426] LustreError: 152538:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.837091] LustreError: 152538:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4861.268426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.754380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4867.325949] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4878.160977] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4885.965881] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4896.200517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4896.849311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4903.821614] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4906.105680] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:161) [ 4912.216973] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4917.402192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:170 to 0x280000401:193) [ 4936.903364] Lustre: lustre-MDT0001: evicting a2f3330e-a33c-4c91-8247-37b4f27a8ecc (at 192.168.204.32@tcp) by administrative request [ 4936.909685] Lustre: Skipped 1 previous similar message [ 4967.905721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4967.920282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4967.946054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4968.933340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4968.946297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4968.973586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4974.087620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4974.105494] Lustre: Skipped 2 previous similar messages [ 4979.172560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4980.191216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4980.473352] Lustre: server umount lustre-OST0000 complete [ 4984.806364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4984.816746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4984.851369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4984.859916] Lustre: Skipped 1 previous similar message [ 4989.410197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.930546] Lustre: server umount lustre-MDT0000 complete [ 4994.410171] LustreError: 152517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123592 with bad export cookie 1680847950878930737 [ 4994.433421] LustreError: 152517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4994.436073] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4994.796907] Lustre: server umount lustre-MDT0001 complete [ 5006.907234] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5009.756584] Key type lgssc unregistered [ 5010.146783] LNet: 156618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5010.165891] LNetError: 156618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5010.182695] LNet: Removed LNI 192.168.204.132@tcp [ 5011.057670] Key type .llcrypt unregistered [ 5011.060696] Key type ._llcrypt unregistered [ 5089.751822] Key type ._llcrypt registered [ 5089.753978] Key type .llcrypt registered [ 5089.888395] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5090.644189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5090.740892] alg: No test for adler32 (adler32-zlib) [ 5091.878371] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 5092.120783] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5093.807605] Key type lgssc registered [ 5095.458308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5105.992850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5106.008411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.437410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5112.437657] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5117.930667] LustreError: 159110:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.973756] LustreError: 159110:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5122.278965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.799214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5127.352887] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5136.452561] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5143.881275] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5150.912411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5151.280765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5152.329634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 5158.558140] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5166.725869] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5167.657190] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:72 to 0x280000400:193) [ 5177.826276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.831647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5177.863310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5177.872977] Lustre: Skipped 1 previous similar message [ 5182.949619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5183.274379] Lustre: server umount lustre-OST0000 complete [ 5188.065240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5188.072857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5188.078231] Lustre: Skipped 2 previous similar messages [ 5188.099121] Lustre: Skipped 2 previous similar messages [ 5192.996818] Lustre: server umount lustre-MDT0000 complete [ 5193.185053] LustreError: 159111:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5197.082067] LustreError: 159092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123795 with bad export cookie 1976896207139627200 [ 5197.114438] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5197.560314] Lustre: server umount lustre-MDT0001 complete [ 5208.318369] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5211.010247] Key type lgssc unregistered [ 5211.272964] LNet: 162656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5211.294589] LNetError: 162656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5211.310327] LNet: Removed LNI 192.168.204.132@tcp [ 5212.172309] Key type .llcrypt unregistered [ 5212.175118] Key type ._llcrypt unregistered [ 5239.555305] Key type ._llcrypt registered [ 5239.559283] Key type .llcrypt registered [ 5239.697401] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5240.917303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5240.983972] alg: No test for adler32 (adler32-zlib) [ 5242.154898] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 5242.413424] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5244.151199] Key type lgssc registered [ 5245.645662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5261.037815] Lustre: DEBUG MARKER: == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 06:50:57 (1789123857) [ 5276.733370] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 5288.525671] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5330.860696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5330.880807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5332.171723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5332.205260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5332.316751] Lustre: lustre-MDT0000: new disk, initializing [ 5332.390546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5332.409400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5337.848461] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5348.302648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.446742] Lustre: 166921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5348.532651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5348.537106] Lustre: Skipped 1 previous similar message [ 5348.611440] Lustre: lustre-MDT0001: new disk, initializing [ 5348.714978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5348.771570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5348.785909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5354.733455] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5366.147284] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5373.049199] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5381.735033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5382.002792] Lustre: lustre-OST0000: new disk, initializing [ 5382.010386] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5382.021053] Lustre: 168890:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5382.124450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5390.021773] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5390.405161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5390.417413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5390.577670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5400.254363] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5411.302023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5411.303308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5411.318781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5416.396322] Lustre: server umount lustre-OST0000 complete [ 5420.513203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5420.526331] LustreError: Skipped 1 previous similar message [ 5420.530221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5420.544278] Lustre: Skipped 1 previous similar message [ 5420.559466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5420.569279] Lustre: Skipped 1 previous similar message [ 5421.537619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.574779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.583937] Lustre: Skipped 1 previous similar message [ 5426.667606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5426.678744] Lustre: Skipped 1 previous similar message [ 5427.101794] Lustre: server umount lustre-MDT0000 complete [ 5431.402265] LustreError: 166915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124029 with bad export cookie 12152824503809972335 [ 5431.404969] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5431.424240] LustreError: 166915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5431.873246] Lustre: server umount lustre-MDT0001 complete [ 5448.298975] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5452.518622] Key type lgssc unregistered [ 5452.986426] LNet: 170801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5452.993786] LNetError: 170801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5454.054717] LNet: Removed LNI 192.168.204.132@tcp [ 5455.458621] Key type .llcrypt unregistered [ 5455.463853] Key type ._llcrypt unregistered [ 5486.157741] Key type ._llcrypt registered [ 5486.159841] Key type .llcrypt registered [ 5486.243912] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5487.120095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5487.172105] alg: No test for adler32 (adler32-zlib) [ 5488.341482] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 5488.699209] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5490.384705] Key type lgssc registered [ 5492.040601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5502.278959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5502.318993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.822453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5508.927512] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5513.196285] LustreError: 172216:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5513.230301] LustreError: 172216:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5518.308551] LustreError: 172215:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.419443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.893600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5523.690683] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5533.763955] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5539.827067] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5546.853662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5547.287100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5549.370848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5555.608393] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5564.410577] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5571.799134] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5571.829612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.857194] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5571.887476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5571.922775] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5573.094512] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5573.110669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5573.125744] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5573.131694] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5573.138802] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5574.185639] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5574.205947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5574.213848] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5574.217918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5574.225129] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5575.058526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5576.074255] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.177638] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5577.182831] Lustre: Skipped 2 previous similar messages [ 5577.205137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.216494] Lustre: Skipped 2 previous similar messages [ 5577.223177] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5577.228443] Lustre: Skipped 2 previous similar messages [ 5577.241368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5577.249580] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5577.262535] Lustre: Skipped 2 previous similar messages [ 5578.088070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.144728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5579.915392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5581.142598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.232197] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5582.244861] Lustre: Skipped 4 previous similar messages [ 5582.264905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.286812] Lustre: Skipped 4 previous similar messages [ 5582.297041] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5582.311503] Lustre: Skipped 4 previous similar messages [ 5582.317271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5582.332532] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5582.343629] Lustre: Skipped 4 previous similar messages [ 5583.379678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5584.597484] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5584.604163] LustreError: 172235:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5584.626490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5585.505399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5586.299318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5587.132504] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5588.218966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5589.151399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5590.202428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.426644] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5591.435534] Lustre: Skipped 8 previous similar messages [ 5591.452788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5591.478774] Lustre: Skipped 8 previous similar messages [ 5591.491598] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5591.497877] Lustre: Skipped 8 previous similar messages [ 5591.504535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5591.521073] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5591.534647] Lustre: Skipped 8 previous similar messages [ 5592.783434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.076487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5595.090648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5596.296739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5597.545536] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5598.623574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5599.446905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5600.418549] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5601.280594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5602.197060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5603.119510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5604.107378] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.080448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5605.941157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5606.841121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5607.957521] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5607.963179] Lustre: Skipped 15 previous similar messages [ 5607.984596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.993725] Lustre: Skipped 15 previous similar messages [ 5607.997471] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5608.002989] Lustre: Skipped 15 previous similar messages [ 5608.006920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5608.022954] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5608.030191] Lustre: Skipped 15 previous similar messages [ 5609.159761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5610.305225] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5611.540405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5612.978216] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5614.084552] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5614.095491] LustreError: 172235:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5614.102226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5615.119774] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5616.246137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5617.358284] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5618.451182] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5619.300280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5620.189935] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5621.119631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.086594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.897324] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5623.873537] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5624.865070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5625.996247] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5627.045946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5628.030230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.052633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5629.848331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5630.693244] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5631.626844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5632.585990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5633.450541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5634.322110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5635.261094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5636.204598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5637.429061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5638.568605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5639.410837] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.372811] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5640.381954] Lustre: Skipped 31 previous similar messages [ 5640.400339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5640.418279] Lustre: Skipped 31 previous similar messages [ 5640.423978] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5640.430408] Lustre: Skipped 31 previous similar messages [ 5640.443204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5640.465196] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5640.473922] Lustre: Skipped 31 previous similar messages [ 5641.630286] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5642.816508] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5644.088567] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5645.091388] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5646.105365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5647.121749] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5648.078323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5649.410500] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5650.762154] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5651.963615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5653.265525] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5654.252125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5655.225050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5656.161421] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5657.171108] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5658.195758] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5659.505423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5660.573915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5661.632102] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5662.512743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5663.769292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5665.055910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5666.176264] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5667.064169] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.023790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5668.980551] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5670.222341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5671.194679] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5672.073759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5673.050924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5674.176698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5675.259327] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5676.300299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5677.369710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5678.139840] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5679.230503] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5680.161392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5681.582587] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5682.572772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5683.469395] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5684.395419] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5685.448672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5686.473636] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5687.277794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5688.196632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.076486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5689.719211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5690.603662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5691.613339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5692.592397] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5693.719566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5694.591226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5695.742938] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5696.813757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5697.745142] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5698.840888] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5699.777698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5700.780591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5701.712519] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5702.857796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5703.780219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.927474] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5704.934210] Lustre: Skipped 61 previous similar messages [ 5704.947218] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5704.954744] Lustre: Skipped 61 previous similar messages [ 5704.957811] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5704.962184] Lustre: Skipped 61 previous similar messages [ 5704.965440] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5704.973605] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5704.978351] Lustre: Skipped 61 previous similar messages [ 5706.131451] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5707.085940] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.162730] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.102899] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5709.936801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5710.805589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5711.904090] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.033138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.958936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5714.940951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5715.920361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.957724] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5717.845471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5718.883529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5719.653908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5720.501086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5721.302846] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5722.288420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5723.333101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5724.206174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5725.113762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.045956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5726.965473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5727.905534] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5727.922199] LustreError: 172235:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5727.936336] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5728.782338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5729.643257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5730.532696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5731.565874] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5732.567732] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5733.530615] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5734.632779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5735.586640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5736.542391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5737.474512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5738.406402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5739.466622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5740.525540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5741.686932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5742.648824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5743.636909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5744.694555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5745.549221] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5746.371651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5747.455929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5748.453876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5749.471380] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5750.601637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5751.560542] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5752.689075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5753.660523] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5754.633176] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5755.590667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5756.520870] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5757.361742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5758.232736] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.066274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.033880] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5760.890609] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5761.896054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5762.729423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5763.637110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5764.584667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5765.752619] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5766.666266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5767.624110] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5768.657923] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5769.597332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5770.740475] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5771.994043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5772.832796] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5773.900464] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5775.054754] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5776.088105] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5777.045776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.147093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5778.982330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5779.930705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5780.788646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5781.655387] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5782.451441] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5783.456819] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5784.571428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5785.642622] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5786.703363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5787.965582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5789.229897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5790.463993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5791.326486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5792.636412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5793.584970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5794.662683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5795.717138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5796.912720] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5798.096353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5799.183352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5800.090353] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5801.349073] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5802.414149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5803.397733] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5804.451446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5805.398170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5806.393063] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5807.501185] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5808.515919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5809.666728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5810.683274] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5811.571520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5812.519552] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5813.467442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5814.377074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5815.463066] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5816.491184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5817.797990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5819.183911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5820.235425] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5821.476304] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5822.591468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5823.644189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5824.624850] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5825.736344] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5826.750470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5827.917589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5829.073867] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5830.549756] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5831.816318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.029026] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5833.034900] Lustre: Skipped 125 previous similar messages [ 5833.046183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5833.063129] Lustre: Skipped 125 previous similar messages [ 5833.067404] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5833.075152] Lustre: Skipped 125 previous similar messages [ 5833.083058] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5833.093988] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5833.098731] Lustre: Skipped 125 previous similar messages [ 5834.215929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5835.253885] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5836.577218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5837.699918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5838.842426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5839.768477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5841.494136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5842.884919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5844.302566] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5845.536978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5846.351564] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5847.885666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5849.791873] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5851.181371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5852.641121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5854.037891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5855.017408] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5856.289445] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5857.287143] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5858.307100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5859.522206] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5860.791667] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5861.928800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5863.035121] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5864.423914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5865.519838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5866.726856] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5867.808415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5868.775385] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5869.718300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5870.701432] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5871.745809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5872.602676] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5873.627711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5874.745422] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5875.711470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5876.657524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5877.689691] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5878.805043] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5879.833088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5880.657219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5881.477813] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5882.493357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5883.511697] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5884.640929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5885.738779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5886.734672] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5887.960595] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5889.089238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5890.195329] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5891.190435] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5892.220735] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5893.229572] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5894.186815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.050487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5895.981529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5896.858932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5897.912715] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5898.811909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5899.699040] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5900.479449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5901.547125] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.415875] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5903.322604] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5904.602565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5905.732130] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5905.739977] LustreError: 172235:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5905.750761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5906.723208] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5908.063610] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5909.272476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5910.505039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.679437] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5912.691162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5913.807844] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 5913.816866] LustreError: 172235:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 5913.828689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5914.791905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5915.768497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5916.740190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5917.585635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5918.553790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.257479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5919.989446] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5920.718631] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5921.614452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5922.546539] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5923.550907] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5924.494393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5925.318273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.080780] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5926.920147] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5927.810980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5928.756483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5929.867896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5930.986952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5931.941390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5932.885714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5933.879859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5934.968251] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5936.327393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5937.477254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5938.766905] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5939.740219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5940.820178] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5941.780722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5942.744165] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5943.830455] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5945.078760] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.036823] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5946.913144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5947.847292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5948.795162] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5949.723399] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5950.811747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5951.693576] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5952.665904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5953.581520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5954.569153] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5955.615079] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5956.610307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5957.611936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5958.565160] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5959.792101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5960.922164] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5961.984368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5962.946527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.905132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5964.906157] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5965.733611] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5966.810126] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5967.677170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5968.509948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5969.449311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5970.333929] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5971.250872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5972.356857] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5973.358908] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5974.340426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5975.191328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5976.279855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5977.200299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5978.319977] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5979.196807] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.067871] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5980.970569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5981.826454] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5982.675008] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5983.676293] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5984.659328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5985.653115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5986.839824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5988.005919] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5989.236067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5990.358894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5991.605866] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.705909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5993.759252] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5994.669246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5995.660983] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5996.703420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5997.831858] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5998.899637] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6000.043889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6001.139700] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.101830] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6002.900956] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6003.990045] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6005.097550] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6006.219339] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6007.632648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6008.837188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6009.943799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6011.025161] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6012.098851] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6013.061785] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6014.272544] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6015.661726] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6017.139039] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6018.429189] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6019.421333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6020.414660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6021.459460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6022.881460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6024.212629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6025.383300] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6026.621050] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6027.638271] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6029.086643] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6030.634628] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6031.904302] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6033.079862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6034.081862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6035.435060] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6036.905647] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6038.393456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6039.389666] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6040.442891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6041.530472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6042.485315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6043.562809] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6044.799394] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6046.026114] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6047.019472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.184535] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6049.318868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6050.575494] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6051.632107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6052.634151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6053.475275] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6054.486717] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6055.462784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6056.580316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6057.626465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6058.483882] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6059.523571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6060.433092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6061.503452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6062.357556] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6063.382177] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6064.235757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6065.200081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.042468] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6066.871714] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6067.706747] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6068.585124] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6069.570828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6070.651167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6071.722145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6072.729348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6073.652772] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6074.639652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6075.675575] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6076.541375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6077.424897] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6078.415140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6079.321075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6080.352307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6081.359041] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6082.318995] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6083.199879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.134471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6085.142881] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6086.101137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.242993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6088.344151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.208215] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6089.213403] Lustre: Skipped 242 previous similar messages [ 6089.228813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6089.237314] Lustre: Skipped 242 previous similar messages [ 6089.240989] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6089.245463] Lustre: Skipped 242 previous similar messages [ 6089.250132] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6089.257677] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6089.265559] Lustre: Skipped 242 previous similar messages [ 6090.325477] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.137879] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6091.970301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6092.819854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6093.696214] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6094.613062] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6095.760582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6096.625279] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6097.528776] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6098.616816] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6099.494212] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6100.487289] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.340183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6102.182638] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6103.061487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.082640] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6104.972793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.960195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6106.792582] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.771218] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6108.665662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6109.633947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6110.578764] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.704926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6112.876452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6113.966376] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6114.969944] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6115.971188] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.979876] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6117.913346] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6119.091761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6120.105827] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.069470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6121.964417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.974832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6124.009290] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6125.256829] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6126.256693] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6127.383639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6128.358708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6129.416317] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6130.221790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6131.424705] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6132.450841] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6133.437845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6134.808232] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6135.737104] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6136.859361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6137.870696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6138.680100] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6139.563690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6140.846179] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6142.227723] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6143.532062] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6143.550143] LustreError: 172235:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6143.550307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6144.543630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6145.690596] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6146.965368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6147.870517] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6148.805889] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6149.770374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6150.954913] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6152.201299] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6153.297653] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6154.193335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6155.295613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6156.283374] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6157.345349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6158.337727] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6159.344152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6160.483139] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6161.557980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6162.472391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.460201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6164.437678] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6165.299588] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.173335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6166.988673] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6167.899904] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6168.718038] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6169.548152] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6170.348925] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6171.303743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6172.254414] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6173.250292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.141664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6175.087790] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6176.289928] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6177.272423] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6178.583092] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6179.747868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6180.534340] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6181.328652] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6182.060332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.040651] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6183.860594] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6184.824920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6185.773512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6186.550847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6187.364954] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6188.400662] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6189.388704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6190.564137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6191.529811] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.449668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6193.330543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.092357] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6194.818728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6195.769070] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6196.593545] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6197.633030] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6198.793024] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6199.891962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6200.805123] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6201.696341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6202.534931] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6203.338458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6204.185620] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6205.086722] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.001332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.840683] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6207.627832] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6208.581655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6209.279151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6210.156553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6211.025657] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6212.142952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6213.181728] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6214.379626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6215.451543] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6216.656521] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6217.751384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6218.765337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6219.834001] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6220.747375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6221.868815] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6222.965547] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6223.886565] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6225.043368] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6226.163095] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.006704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6227.883812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6228.839784] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6229.717591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6230.625573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6231.468266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6232.471462] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6233.343472] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6234.294405] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6235.272817] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6236.281933] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6237.451924] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.490145] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6239.341297] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6240.313639] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6241.380233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6242.579586] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6243.720645] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6244.872384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6245.960183] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6247.059569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6248.224365] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6249.359946] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6250.458483] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6251.423868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6252.535495] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6253.955452] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6255.419471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6256.715433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6258.620375] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6260.163560] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6261.589573] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6263.277624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6264.409966] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.807201] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6267.496233] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6268.668135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6269.839320] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6269.858657] LustreError: 172235:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6269.878163] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6271.005658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6272.435426] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6273.717266] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6275.196641] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6276.827980] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6278.132328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6279.867668] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6281.768093] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6283.191044] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6284.929369] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6286.290972] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6287.504753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6288.590361] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6289.570389] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6290.676743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6291.691822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6292.833568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6294.013531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6295.287744] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6296.558909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6297.704466] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6298.885352] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6300.083855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6301.199775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6302.229993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6303.424704] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6304.463330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6305.716865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6306.822739] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6308.058812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6309.219471] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6310.422213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6311.762174] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6313.004633] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6314.577191] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6315.909292] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.051632] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6317.974456] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6318.858561] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6319.673481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6320.832026] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6321.800325] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6322.835107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6323.740097] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6324.684181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6325.516301] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6326.421682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6327.291184] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6328.226186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6329.151181] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6330.136457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.059470] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6331.962942] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6332.901712] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6333.923775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6334.817270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6335.919273] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6336.906967] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6337.980237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6339.018990] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6340.140812] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6341.195765] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6342.364172] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6343.494958] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6344.574961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6345.638384] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6346.714598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6347.672393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6348.925487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.003820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6350.957135] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6352.012912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.051200] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6353.981282] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.040987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6355.866801] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6356.735473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6357.688629] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6358.472947] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6359.346838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.257434] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6361.150088] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6362.185420] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6363.101799] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6364.193417] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6365.396239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6366.434900] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6367.422170] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6368.486488] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6369.264926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6370.259927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6371.178868] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.042820] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6372.861522] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6374.003554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6375.214541] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.241277] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6377.337493] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6378.412869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6379.611265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.697960] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6381.632230] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6382.629961] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.539480] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6384.742416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6385.778798] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6386.768731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6387.899330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6389.017833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6390.060696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6391.285822] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6392.345769] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6393.464295] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6394.641061] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6395.719854] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6396.623476] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.806267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6398.728366] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6399.650932] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6400.482993] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6401.484783] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.371791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6403.404510] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6404.485589] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6405.686312] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6406.754681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6407.901696] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6408.906505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6409.948529] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6410.954449] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6411.976654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6413.141591] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6414.318695] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6414.333993] LustreError: 172235:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6414.365076] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6415.535402] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6416.666144] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6417.643948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6418.856527] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6419.872590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6420.838605] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6421.805777] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6422.682648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6423.605766] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6424.579487] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6425.725415] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6426.902546] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6427.868831] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6428.859138] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6429.827514] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6430.747865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6431.486186] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6432.454806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6433.256580] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.179624] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6434.921098] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6435.809371] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6436.891574] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6437.642096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6438.613448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6439.549740] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6440.468826] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6441.433828] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6442.501074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6443.605806] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6444.563520] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6445.606775] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6446.655457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6447.774383] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6449.064269] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6450.314571] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6451.371416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6452.586412] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6453.590442] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6454.627646] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6455.675473] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6456.675140] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6457.639321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6458.718865] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6459.639852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6460.522526] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6461.419912] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6462.347524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6463.387585] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6464.351238] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6465.383457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6466.347128] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6467.452332] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6468.481687] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6469.418936] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6470.402257] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6471.369671] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6472.394151] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6473.344773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6474.461133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6475.606052] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6476.711745] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6477.737362] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6478.907970] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6479.916847] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6481.143689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6482.197987] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.277389] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 [ 6483.287055] LustreError: 172235:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 [ 6483.297742] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6484.295433] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.362089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6486.665107] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6488.022307] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.171793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6490.072794] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6491.055844] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6492.063280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6493.122136] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6494.065276] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6495.059505] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.057569] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6496.958254] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6497.855759] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6498.586779] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6499.509708] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6500.570501] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.520278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6502.372852] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6503.415457] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6504.347080] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6505.353416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6506.209512] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6507.110553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6508.287757] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6509.298087] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6510.336731] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6511.244131] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6512.325428] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6513.238272] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6514.362978] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6515.348316] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.496914] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6517.501075] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6518.568859] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6519.477934] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6520.584448] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6521.420838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6522.405281] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6523.464743] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6524.633479] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6525.618711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6526.837741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.839089] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6529.085862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6530.098877] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6531.109323] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.569303] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6533.705800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6534.893943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6535.850746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6536.765800] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6537.738411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6538.861648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6539.822116] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6540.752239] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6541.606396] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6542.677067] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6543.661654] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6544.743333] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6546.067602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6547.240306] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6548.156318] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6549.209761] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6550.220616] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6551.379793] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6552.556499] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6553.522555] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6554.459393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6555.590392] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6556.803658] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6557.746710] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.996849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6560.008937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6561.105029] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6562.084036] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.156969] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6564.215224] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6565.121892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6566.019119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6567.093869] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6568.050235] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6569.204246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6570.070926] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6571.215635] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6572.331894] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6573.402648] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6574.469553] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6575.402753] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6576.355042] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6577.528921] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6578.678338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6579.764892] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6580.939430] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6581.895133] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6582.814259] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6583.754665] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6584.937467] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6585.983155] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6587.172706] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6588.231248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6589.274215] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6590.434862] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6591.669719] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6592.681792] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6593.573824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6594.514046] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6595.513330] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6596.414943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.436246] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6598.445443] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6599.542054] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6600.595791] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6601.555605] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6601.563865] Lustre: Skipped 491 previous similar messages [ 6601.585769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.597218] Lustre: Skipped 491 previous similar messages [ 6601.601706] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6601.610193] Lustre: Skipped 491 previous similar messages [ 6601.614081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6601.625630] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6601.632109] Lustre: Skipped 491 previous similar messages [ 6602.708436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6603.584948] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6604.415838] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6605.307411] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6606.210265] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6607.073416] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6607.853971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6608.797810] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.696945] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7058.699331] LustreError: 174039:0:(ofd_dev.c:1794:ofd_create_hdl()) lustre-OST0000: unable to precreate: rc = -28 [ 7058.709661] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [ 7058.722650] LustreError: 172235:0:(osp_precreate.c:655:osp_precreate_send()) Skipped 2 previous similar messages [ 7791.079345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7791.094986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7791.108252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7791.114243] Lustre: Skipped 10 previous similar messages [ 7791.145365] LustreError: Skipped 4 previous similar messages [ 7796.200113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7796.225074] Lustre: Skipped 2 previous similar messages [ 7797.397634] Lustre: server umount lustre-OST0000 complete [ 7802.341410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7802.361716] LustreError: Skipped 3 previous similar messages [ 7802.384960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7806.434335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7806.442631] Lustre: Skipped 1 previous similar message [ 7809.213407] Lustre: server umount lustre-MDT0000 complete [ 7811.553100] LustreError: 173211:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7811.578410] LustreError: 173211:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7815.592171] LustreError: 172197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126413 with bad export cookie 655048645506227598 [ 7815.611260] LustreError: 172197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7815.614789] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7816.578891] Lustre: server umount lustre-MDT0001 complete [ 7831.610137] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7835.244536] Key type lgssc unregistered [ 7835.741310] LNet: 227531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7835.755957] LNetError: 227531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7835.774926] LNet: Removed LNI 192.168.204.132@tcp [ 7837.188316] Key type .llcrypt unregistered [ 7837.193875] Key type ._llcrypt unregistered [ 7868.066362] Key type ._llcrypt registered [ 7868.076233] Key type .llcrypt registered [ 7868.213863] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7869.402314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7869.436423] alg: No test for adler32 (adler32-zlib) [ 7870.489373] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 7870.692201] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 7872.367217] Key type lgssc registered [ 7873.708715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7883.377895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7883.436863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7884.997102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7890.551884] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7894.501560] LustreError: 228952:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7894.537624] LustreError: 228952:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7899.616983] LustreError: 228953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7900.121245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7900.567638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7905.178912] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7914.966229] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7922.306314] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7930.777559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.081207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7938.719990] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7949.468903] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7954.705990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49664 to 0x280000401:49695) [ 7960.335439] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7962.782172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7962.809745] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 7962.824194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7962.857069] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7962.857766] LustreError: 228972:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 sleeping for 20000ms [ 7966.180264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7966.190758] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7966.223460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7967.018348] Lustre: lustre-OST0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 7970.276146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7970.276463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.109723] LustreError: 231445:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7972.324949] Lustre: server umount lustre-OST0000 complete [ 7982.935915] LustreError: 228972:0:(osp_precreate.c:1268:osp_precreate_thread()) cfs_fail_timeout id 2107 awake [ 8009.291759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8009.789559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8010.830891] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8010.846835] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8016.488609] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8020.072655] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8020.086975] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8025.439708] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8027.433572] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8035.150389] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9870c6db1000.ost_server_uuid 50 [ 8037.026139] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9870c6db1000.ost_server_uuid in FULL state after 0 sec [ 8042.466695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8042.480353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8042.503553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8042.515679] Lustre: Skipped 1 previous similar message [ 8043.472147] Lustre: server umount lustre-OST0000 complete [ 8049.123575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8049.137872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8049.162746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8054.112097] Lustre: server umount lustre-MDT0000 complete [ 8056.298056] LustreError: 228952:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8056.320356] LustreError: 228952:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8058.442827] LustreError: 228933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126656 with bad export cookie 3942216816488305349 [ 8058.443916] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8058.463299] LustreError: 228933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8068.304458] Lustre: server umount lustre-MDT0001 complete [ 8081.954411] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 8084.975490] Key type lgssc unregistered [ 8085.347690] LNet: 234165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8085.361242] LNetError: 234165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8086.457106] LNet: Removed LNI 192.168.204.132@tcp [ 8087.656915] Key type .llcrypt unregistered [ 8087.660282] Key type ._llcrypt unregistered [ 8126.088470] Key type ._llcrypt registered [ 8126.096364] Key type .llcrypt registered [ 8126.278592] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 8127.997105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8128.048284] alg: No test for adler32 (adler32-zlib) [ 8129.142307] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 8129.447684] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 8131.183387] Key type lgssc registered [ 8132.511516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8143.514560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8143.538537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.066340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8150.836993] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8154.598158] LustreError: 235896:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8154.628529] LustreError: 235896:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8159.714391] LustreError: 235897:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8161.070572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8161.465291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8166.342092] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8170.181411] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8177.757664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8178.138217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8185.535787] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8189.498303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49697 to 0x280000401:49727) [ 8196.526204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8196.761569] Lustre: lustre-OST0001: new disk, initializing [ 8196.765545] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8196.777718] Lustre: 238207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8196.930609] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8202.870409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8202.890612] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8203.041382] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8204.226674] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8216.946984] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8226.198103] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8243.168482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8243.174801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8243.201324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8244.199595] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8244.202683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8244.221710] Lustre: Skipped 2 previous similar messages [ 8244.239979] Lustre: Skipped 3 previous similar messages [ 8247.704357] Lustre: server umount lustre-MDT0000 complete [ 8249.322597] LustreError: 236625:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8249.366471] LustreError: 236625:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8251.773626] LustreError: 235878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126849 with bad export cookie 14058758247506420618 [ 8251.785243] LustreError: 235878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8251.787353] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8252.306183] Lustre: server umount lustre-MDT0001 complete [ 8267.100485] Lustre: server umount lustre-OST0000 complete [ 8280.981108] Lustre: server umount lustre-OST0001 complete [ 8305.770669] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 8317.647411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8318.212846] LustreError: 241318:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8318.300090] Lustre: mylustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8323.554890] LustreError: 241319:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8323.602172] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8333.210321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8333.703973] Lustre: mylustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8339.060110] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8342.068467] Lustre: 242461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8349.750915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8350.067519] Lustre: mylustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8358.143513] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8359.292035] LustreError: 242814:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8359.335591] LustreError: 242814:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8364.421683] Lustre: mylustre-OST0000: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49729 to 0x280000401:49759) [ 8369.600803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8375.322206] Lustre: mylustre-OST0001: new connection from mylustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8378.284645] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8386.347669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8391.178430] Lustre: 244331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8405.474223] LustreError: mylustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8405.483355] Lustre: mylustre-MDT0000-osp-MDT0001: Connection to mylustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8405.500403] Lustre: mylustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8408.826810] Lustre: server umount mylustre-MDT0000 complete [ 8411.111529] LustreError: 243039:0:(ldlm_lib.c:1228:target_handle_connect()) mylustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8411.132974] LustreError: 243039:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 8412.754832] LustreError: 241294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127010 with bad export cookie 14058758247506423425 [ 8412.756729] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8412.764178] LustreError: 241294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8413.141797] Lustre: server umount mylustre-MDT0001 complete [ 8427.581500] Lustre: server umount mylustre-OST0000 complete [ 8441.409592] Lustre: server umount mylustre-OST0001 complete [ 8463.873992] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 8474.281227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8474.631908] LustreError: 246644:0:(ldlm_lib.c:1228:target_handle_connect()) tfs-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8474.705632] Lustre: tfs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8474.710774] Lustre: Skipped 1 previous similar message [ 8479.387377] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8489.185375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.373000] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8497.871556] Lustre: 247785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8504.548485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8511.372814] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8519.305640] Lustre: tfs-OST0000: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49761 to 0x280000401:49791) [ 8521.640074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8527.364781] Lustre: tfs-OST0001: new connection from tfs-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [ 8529.415160] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8538.250764] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8542.790231] Lustre: 249657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8550.883332] LustreError: tfs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8550.888536] Lustre: tfs-MDT0000-osp-MDT0001: Connection to tfs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.906623] Lustre: Skipped 3 previous similar messages [ 8550.911720] Lustre: tfs-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.923840] Lustre: Skipped 3 previous similar messages [ 8552.689822] Lustre: server umount tfs-MDT0000 complete [ 8552.939729] LustreError: 246644:0:(ldlm_lib.c:1228:target_handle_connect()) tfs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8552.954276] LustreError: 246644:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 8556.336515] LustreError: 246619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127154 with bad export cookie 14058758247506425315 [ 8556.352264] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.358789] LustreError: 246619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8556.921891] Lustre: server umount tfs-MDT0001 complete [ 8571.087723] Lustre: server umount tfs-OST0000 complete [ 8574.415199] Lustre: 235161:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789127156/real 1789127156] req@ffff983b15ec8f00 x1876035347102080/t0(0) o400->tfs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789127172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8574.439443] Lustre: tfs-MDT0001-lwp-OST0001: Connection to tfs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8574.447613] Lustre: Skipped 3 previous similar messages [ 8575.192770] Lustre: server umount tfs-OST0001 complete [ 8601.639682] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 8612.394658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8613.029481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8613.036058] Lustre: Skipped 3 previous similar messages [ 8618.003974] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8628.774307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8634.308208] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8637.516995] Lustre: 253054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8645.641517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8652.883119] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8657.346270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:49761 to 0x280000401:49823) [ 8663.358698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8668.668317] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [ 8670.470743] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8679.073012] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8683.091465] Lustre: 254926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8691.018792] Lustre: DEBUG MARKER: == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 07:48:06 (1789127286) [ 8709.608083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8709.612404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8709.632947] Lustre: Skipped 2 previous similar messages [ 8709.651712] Lustre: Skipped 3 previous similar messages [ 8714.723852] LustreError: 251905:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.754430] LustreError: 251905:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 8714.891192] Lustre: server umount lustre-MDT0000 complete [ 8719.021432] LustreError: 251886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127316 with bad export cookie 14058758247506426974 [ 8719.033830] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8719.034381] LustreError: 251886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8719.845927] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8719.851085] Lustre: Skipped 1 previous similar message [ 8725.943582] Lustre: server umount lustre-MDT0001 complete [ 8737.168049] Lustre: server umount lustre-OST0000 complete [ 8756.194296] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8756.513459] Lustre: server umount lustre-OST0001 complete [ 8764.116814] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 8776.798984] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 8820.691391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.929209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8820.978954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8821.103934] Lustre: lustre-MDT0000: new disk, initializing [ 8821.203216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 188-180 [ 8821.213645] Lustre: Skipped 3 previous similar messages [ 8821.236249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8826.214978] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8837.850050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.095400] Lustre: 258113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8838.187426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8838.197242] Lustre: Skipped 1 previous similar message [ 8838.335483] Lustre: lustre-MDT0001: new disk, initializing [ 8838.536941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8838.551733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8843.830452] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8855.254146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8855.527100] Lustre: lustre-OST0000: new disk, initializing [ 8855.543241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8855.554956] Lustre: 259695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8856.862402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8856.874459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8857.032282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8863.029810] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8875.186740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8875.434346] Lustre: lustre-OST0001: new disk, initializing [ 8875.440967] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8875.463809] Lustre: 260567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8877.455240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8877.468537] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8877.596599] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8883.074723] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 8897.555194] Lustre: DEBUG MARKER: == conf-sanity test 104b: Mount uses last flock argument ========================================================== 07:51:33 (1789127493) [ 8914.440630] Lustre: DEBUG MARKER: == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 07:51:50 (1789127510) [ 8933.863094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8933.868106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8933.880804] LustreError: Skipped 1 previous similar message [ 8933.882821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8933.882832] Lustre: Skipped 2 previous similar messages [ 8933.927441] Lustre: Skipped 4 previous similar messages [ 8935.554756] Lustre: server umount lustre-OST0000 complete [ 8941.024634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8945.904624] Lustre: server umount lustre-MDT0000 complete [ 8949.998816] LustreError: 258106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127547 with bad export cookie 14058758247506428535 [ 8950.020770] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8950.024799] LustreError: 258106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8956.465494] Lustre: server umount lustre-MDT0001 complete [ 8969.532779] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 9042.046946] Lustre: server umount lustre-OST0001 complete [ 9049.974837] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 9069.367662] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 9114.681343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.042251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9115.077272] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9115.186704] Lustre: lustre-MDT0000: new disk, initializing [ 9115.294219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9115.302740] Lustre: Skipped 3 previous similar messages [ 9115.321683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9120.579424] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9132.597617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9132.763747] Lustre: 265419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9132.812600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9132.819741] Lustre: Skipped 1 previous similar message [ 9132.891434] Lustre: lustre-MDT0001: new disk, initializing [ 9133.018340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9133.036849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9138.434784] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9150.408808] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9157.986470] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9168.949691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9169.381129] Lustre: lustre-OST0000: new disk, initializing [ 9169.389721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9169.397726] Lustre: 267388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9170.717686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9170.751135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9170.891814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9177.421601] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9187.651812] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9196.512545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9196.527158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9196.550197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9196.550941] Lustre: Skipped 5 previous similar messages [ 9196.578777] Lustre: Skipped 8 previous similar messages [ 9201.276198] Lustre: server umount lustre-OST0000 complete [ 9211.873365] LustreError: 265429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9211.903661] LustreError: 265429:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [ 9211.968650] Lustre: server umount lustre-MDT0000 complete [ 9215.999187] LustreError: 266214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127813 with bad export cookie 14058758247506429998 [ 9216.015081] LustreError: 266214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9216.020142] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9216.440902] Lustre: server umount lustre-MDT0001 complete [ 9231.807221] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 9235.626527] Key type lgssc unregistered [ 9235.913582] LNet: 269151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9235.919892] LNetError: 269151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9235.946354] LNet: Removed LNI 192.168.204.132@tcp [ 9237.133283] Key type .llcrypt unregistered [ 9237.142044] Key type ._llcrypt unregistered [ 9269.510985] Key type ._llcrypt registered [ 9269.513666] Key type .llcrypt registered [ 9269.666265] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 9271.151619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9271.335859] alg: No test for adler32 (adler32-zlib) [ 9272.439701] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 9272.656299] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 9274.319432] Key type lgssc registered [ 9275.514630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9284.449578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9284.479739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.009102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9291.435376] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9295.338585] LustreError: 270618:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9295.370571] LustreError: 270618:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9300.450693] LustreError: 270619:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9302.221107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9302.783639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9308.134880] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9317.835185] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9325.259482] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9333.508694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9334.057734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9341.550516] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9351.086891] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9352.271748] Lustre: Setting parameter lustre-OST0000.ost.unknown_param=50 in log lustre-OST0000 [ 9355.438623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9355.445787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9355.494609] Lustre: Skipped 1 previous similar message [ 9360.770646] Lustre: server umount lustre-OST0000 complete [ 9365.990179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.992453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9366.009357] Lustre: Skipped 1 previous similar message [ 9366.020500] Lustre: Skipped 1 previous similar message [ 9371.131873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9371.148371] Lustre: Skipped 1 previous similar message [ 9373.964862] Lustre: server umount lustre-MDT0000 complete [ 9375.266310] LustreError: 270615:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9375.302324] LustreError: 270615:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9379.031321] LustreError: 270598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127976 with bad export cookie 12913026952142963753 [ 9379.033094] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9379.039833] LustreError: 270598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9379.654273] Lustre: server umount lustre-MDT0001 complete [ 9393.355861] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 9397.268215] Key type lgssc unregistered [ 9397.642584] LNet: 274209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9397.650836] LNetError: 274209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9397.665622] LNet: Removed LNI 192.168.204.132@tcp [ 9398.732150] Key type .llcrypt unregistered [ 9398.736640] Key type ._llcrypt unregistered [ 9417.117362] Key type ._llcrypt registered [ 9417.121101] Key type .llcrypt registered [ 9417.279351] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 9418.295196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9418.396455] alg: No test for adler32 (adler32-zlib) [ 9419.527308] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 9419.745724] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 9421.416106] Key type lgssc registered [ 9422.677101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9431.747315] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9431.777963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9447.391764] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9452.511552] LustreError: 275403:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 9479.263927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9486.169135] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9494.903385] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9497.503085] Lustre: server umount lustre-OST0000 complete [ 9517.195693] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 9520.136088] Key type lgssc unregistered [ 9520.527038] LNet: 277051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9520.547853] LNetError: 277051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9520.568948] LNet: Removed LNI 192.168.204.132@tcp [ 9521.483691] Key type .llcrypt unregistered [ 9521.487035] Key type ._llcrypt unregistered [ 9562.107726] Key type ._llcrypt registered [ 9562.109674] Key type .llcrypt registered [ 9562.179109] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 9563.252057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9563.353772] alg: No test for adler32 (adler32-zlib) [ 9564.529188] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [ 9564.779264] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 9566.497068] Key type lgssc registered [ 9567.802699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9580.902317] loop0: detected capacity change from 0 to 204800000 [ 9581.064967] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9581.072688] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9581.443559] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9582.439437] loop0: detected capacity change from 0 to 204800000 [ 9582.685641] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9582.711792] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9583.132483] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9584.094655] loop0: detected capacity change from 0 to 204800000 [ 9584.312931] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9584.323822] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9584.411354] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9585.704713] loop0: detected capacity change from 0 to 204800000 [ 9585.899634] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9586.089381] print_req_error: 1 callbacks suppressed [ 9586.089392] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9587.212680] loop0: detected capacity change from 0 to 204800000 [ 9587.251869] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9591.557876] loop0: detected capacity change from 0 to 204800000 [ 9591.588636] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9596.434739] loop0: detected capacity change from 0 to 204800000 [ 9596.459540] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9601.401613] loop0: detected capacity change from 0 to 204800000 [ 9601.419195] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) [ 9606.336753] loop0: detected capacity change from 0 to 204800000 [ 9606.428712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9606.502892] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9606.608450] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9606.617842] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 [ 9606.636072] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 [ 9606.646736] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9606.658231] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9606.682038] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 [ 9606.702357] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9606.718795] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 [ 9606.738107] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 [ 9606.749547] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 [ 9606.761551] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 [ 9606.775380] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 [ 9606.788742] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 [ 9606.816789] Lustre: 280472:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 [ 9613.131189] Lustre: server umount MGS complete [ 9614.502973] loop0: detected capacity change from 0 to 204800000 [ 9614.662778] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.403837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9615.487381] Lustre: lustre-MDT0000: Aborting client recovery [ 9616.494554] loop1: detected capacity change from 0 to 204800000 [ 9616.643535] LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9616.745336] Lustre: 280907:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9616.756584] Lustre: 280907:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 [ 9616.772300] Lustre: 280907:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 [ 9616.790766] Lustre: 280907:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 [ 9616.804228] Lustre: 280907:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9616.826337] Lustre: 280907:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9616.845791] Lustre: 280907:0:(osd_scrub.c:1854:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9616.876663] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9617.111119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9617.163750] Lustre: lustre-MDT0001: Aborting client recovery [ 9618.158726] loop2: detected capacity change from 0 to 204800000 [ 9618.332755] LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9618.465836] Lustre: 280986:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9618.477963] Lustre: 280986:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 [ 9618.498923] Lustre: 280986:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 [ 9618.534788] Lustre: 280986:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9618.548443] Lustre: 280986:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9618.560141] Lustre: 280986:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9618.593821] Lustre: 280986:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 [ 9618.836191] Lustre: Found index 0 for lustre-OST0000, updating log [ 9618.904874] Lustre: 281004:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9619.022416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9619.056437] Lustre: lustre-OST0000: Aborting recovery [ 9620.037322] loop3: detected capacity change from 0 to 204800000 [ 9620.147201] LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9620.242103] Lustre: 281071:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 [ 9620.251935] Lustre: 281071:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 [ 9620.267507] Lustre: 281071:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 [ 9620.288621] Lustre: 281071:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 [ 9620.304109] Lustre: 281071:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 [ 9620.316239] Lustre: 281071:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 [ 9620.327739] Lustre: 281071:0:(osd_scrub.c:1854:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 [ 9620.357691] Lustre: Found index 1 for lustre-OST0001, updating log [ 9620.384494] Lustre: 281076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9620.473683] Lustre: lustre-OST0001: Aborting recovery [ 9625.575226] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9625.608133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x0:6 to 0x0:33) [ 9625.623523] LustreError: 280851:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -108 [ 9625.637246] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000bd0 [ 9625.683431] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000bd0 [ 9627.623162] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5 to 0x2c0000400:33) [ 9627.660369] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:33) [ 9630.301457] LustreError: 280818:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0000: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0000 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0000/identity_upcall, time 2686us: rc = -2 [ 9630.812622] LustreError: 280819:0:(mdt_identity.c:115:mdt_identity_do_upcall()) lustre-MDT0001: error invoking upcall /root/Work/Lustre/R5/lustre-release/lustre/tests/../utils/l_getidentity lustre-MDT0001 0: rc -2; check /proc/fs/lustre/mdt/lustre-MDT0001/identity_upcall, time 1077us: rc = -2 [ 9630.844312] LustreError: 280819:0:(mdt_identity.c:115:mdt_identity_do_upcall()) Skipped 14 previous similar messages [ 9648.097527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9648.097584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9648.099954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9648.145407] Lustre: Skipped 2 previous similar messages [ 9651.157757] Lustre: server umount lustre-MDT0000 complete [ 9652.287470] LustreError: 280801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128250 with bad export cookie 12517766686057363040 [ 9652.306194] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9652.312692] LustreError: 280801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9653.216950] LustreError: 280816:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9653.218476] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9653.219102] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9653.219108] Lustre: Skipped 3 previous similar messages [ 9653.236301] LustreError: 280816:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9653.267729] Lustre: Skipped 2 previous similar messages [ 9657.249101] LustreError: 280816:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9657.252156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9657.273585] LustreError: 280816:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9657.284842] Lustre: Skipped 2 previous similar messages [ 9658.804309] Lustre: server umount lustre-MDT0001 complete [ 9674.209310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9674.446512] Lustre: server umount lustre-OST0000 complete [ 9681.734338] Lustre: server umount lustre-OST0001 complete [ 9691.605245] Lustre: DEBUG MARKER: == conf-sanity test 109a: test lctl clear_conf fsname ==== 08:04:47 (1789128287) [ 9723.081204] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 9733.225116] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 9770.640553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9770.834554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9770.894992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9770.970773] Lustre: lustre-MDT0000: new disk, initializing [ 9771.071824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.076327] Lustre: Skipped 1 previous similar message [ 9771.099468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9775.779857] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9787.073253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9787.179764] Lustre: 285141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9787.236846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9787.240562] Lustre: Skipped 1 previous similar message [ 9787.316407] Lustre: lustre-MDT0001: new disk, initializing [ 9787.443918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9787.511385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9787.523386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9793.260625] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9803.726716] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9809.837637] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9817.208919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9817.548097] Lustre: lustre-OST0000: new disk, initializing [ 9817.551688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9817.556901] Lustre: 287111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9817.647541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9823.322143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9823.331460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9823.455694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9824.361824] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9834.198899] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9839.537279] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [ 9850.894716] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [ 9858.160614] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [ 9910.758911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9910.758956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9910.760816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9910.771826] LustreError: Skipped 1 previous similar message [ 9910.806484] Lustre: Skipped 1 previous similar message [ 9915.873604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9915.887221] Lustre: Skipped 1 previous similar message [ 9916.482451] Lustre: server umount lustre-OST0000 complete [ 9920.485352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9920.501722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9920.526790] Lustre: Skipped 1 previous similar message [ 9926.112112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9926.116729] Lustre: Skipped 3 previous similar messages [ 9926.468745] Lustre: server umount lustre-MDT0000 complete [ 9929.936518] LustreError: 285936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128527 with bad export cookie 12517766686057365749 [ 9929.941109] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9929.953864] LustreError: 285936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9930.310322] Lustre: server umount lustre-MDT0001 complete [ 9938.998569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.689297] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9948.749806] Lustre: server umount MGS complete [ 9960.720494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9961.086178] LustreError: 291049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9961.115723] LustreError: 291062:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [ 9961.186420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.344144] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9966.572486] LustreError: 291050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9971.689037] LustreError: 291049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9974.619257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.906624] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 9988.651549] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9995.103916] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10001.610709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10001.850206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10001.854197] Lustre: Skipped 1 previous similar message [10008.049353] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [10015.097500] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10044.900809] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10044.904027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.915118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10044.923347] Lustre: Skipped 2 previous similar messages [10044.947879] Lustre: Skipped 1 previous similar message [10050.203986] Lustre: server umount lustre-OST0000 complete [10055.136853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10055.158538] Lustre: Skipped 1 previous similar message [10060.206661] Lustre: server umount lustre-MDT0000 complete [10060.259796] LustreError: 291050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10060.295113] LustreError: 291050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10064.267164] LustreError: 291029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128662 with bad export cookie 12517766686057367590 [10064.269053] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10064.281986] LustreError: 291029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10064.910816] Lustre: server umount lustre-MDT0001 complete [10077.848798] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [10080.624531] Key type lgssc unregistered [10080.992970] LNet: 294996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10081.000215] LNetError: 294996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10081.015482] LNet: Removed LNI 192.168.204.132@tcp [10082.074977] Key type .llcrypt unregistered [10082.076772] Key type ._llcrypt unregistered [10119.153855] Key type ._llcrypt registered [10119.156560] Key type .llcrypt registered [10119.265296] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [10133.807371] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [10134.807695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10134.959524] alg: No test for adler32 (adler32-zlib) [10136.075540] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [10136.290260] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [10137.927198] Key type lgssc registered [10139.185869] Lustre: Echo OBD driver; http://www.lustre.org/ [10174.408318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10174.468596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10175.732899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10175.755283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10175.813207] Lustre: lustre-MDT0000: new disk, initializing [10175.916321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10175.949208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10180.907579] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [10191.806896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10191.917209] Lustre: 298959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10191.981258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10191.984720] Lustre: Skipped 1 previous similar message [10192.046621] Lustre: lustre-MDT0001: new disk, initializing [10192.145688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10192.182879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10192.200456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10197.414963] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [10208.197968] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10213.963249] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10220.355890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10220.677825] Lustre: lustre-OST0000: new disk, initializing [10220.684918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10220.696722] Lustre: 300927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10220.801571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10227.825802] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [10229.863452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10229.879629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10230.054054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10237.613249] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10247.248696] Lustre: Setting parameter lustre-MDT0000.mdd.atime_diff=62 in log lustre-MDT0000 [10260.061048] Lustre: Modifying parameter lustre-MDT0000.mdd.atime_diff=63 in log lustre-MDT0000 [10264.348422] Lustre: Setting parameter lustre-client.llite.max_read_ahead_mb=32 in log lustre-client [10271.057763] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_mb=64 in log lustre-client [10317.285131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.294806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10317.307464] Lustre: Skipped 1 previous similar message [10322.409627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10322.414664] Lustre: Skipped 1 previous similar message [10327.526234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10327.540480] Lustre: Skipped 1 previous similar message [10329.055273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10329.235466] Lustre: server umount lustre-OST0000 complete [10335.714629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10335.729937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10335.742079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10337.760573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10339.411212] Lustre: server umount lustre-MDT0000 complete [10343.357867] LustreError: 299753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128941 with bad export cookie 7974373566791044032 [10343.368323] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10343.372287] LustreError: 299753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10343.780775] Lustre: server umount lustre-MDT0001 complete [10353.043211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.568792] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [10363.352131] Lustre: server umount MGS complete [10375.737087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10376.198160] LustreError: 304746:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10376.215250] LustreError: 304746:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10376.227894] LustreError: 304759:0:(qmt_pool.c:1639:qmt_pool_add_rem()) lustre-QMT0000: can't remove lustre-OST0000_UUID pool 'pool1': rc = -22 [10376.276713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10380.815660] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [10381.280368] LustreError: 304747:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10386.401448] LustreError: 304746:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10389.833782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10390.178085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10394.768333] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [10402.608205] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10408.723575] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10415.933741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10416.041236] LustreError: 306572:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.330188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10421.840449] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [10429.455700] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10462.688563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10462.692738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10462.695207] LustreError: Skipped 1 previous similar message [10462.703497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10462.707186] Lustre: Skipped 1 previous similar message [10462.735795] Lustre: Skipped 3 previous similar messages [10464.531422] Lustre: server umount lustre-OST0000 complete [10472.418192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10472.431072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10472.445588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10475.493336] Lustre: server umount lustre-MDT0000 complete [10478.050668] LustreError: 304747:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.081431] LustreError: 304747:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10480.332146] LustreError: 306593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129078 with bad export cookie 7974373566791045880 [10480.334558] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10480.352117] LustreError: 306593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10480.688561] Lustre: server umount lustre-MDT0001 complete [10493.060153] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [10495.963860] Key type lgssc unregistered [10496.272641] LNet: 308699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10496.278557] LNetError: 308699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10497.318377] LNet: Removed LNI 192.168.204.132@tcp [10498.077048] Key type .llcrypt unregistered [10498.078826] Key type ._llcrypt unregistered [10521.531625] Key type ._llcrypt registered [10521.534532] Key type .llcrypt registered [10521.653397] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [10536.604358] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [10537.619974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10537.692651] alg: No test for adler32 (adler32-zlib) [10538.907480] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [10539.249838] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [10541.055410] Key type lgssc registered [10542.474665] Lustre: Echo OBD driver; http://www.lustre.org/ [10571.073576] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [10589.583096] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10589.614479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10590.949200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10591.013818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10591.105811] Lustre: lustre-MDT0000: new disk, initializing [10591.249881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10591.292415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10597.210110] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [10610.061435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10610.387918] Lustre: lustre-OST0000: new disk, initializing [10610.393981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10610.401873] Lustre: Skipped 1 previous similar message [10610.407626] Lustre: 312750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10610.529790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10618.051874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10618.068299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10618.126894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10618.183257] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [11639.777987] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11639.804957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11643.340530] Lustre: server umount lustre-MDT0000 complete [11649.270156] LustreError: 311938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130247 with bad export cookie 15936770094692917826 [11649.285878] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11649.288467] LustreError: 311938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11649.428311] Lustre: server umount lustre-OST0000 complete [11684.023701] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [11699.868477] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [11742.499262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.803291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11742.854166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11742.976689] Lustre: lustre-MDT0000: new disk, initializing [11743.124546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11743.163077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11748.659855] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [11761.005433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11761.158829] Lustre: 317129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11761.221473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11761.227177] Lustre: Skipped 1 previous similar message [11761.297250] Lustre: lustre-MDT0001: new disk, initializing [11761.403749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11761.444886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11761.470379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11766.434896] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [11777.542228] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11784.664610] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11793.310768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11793.805554] Lustre: lustre-OST0000: new disk, initializing [11793.816231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11793.837149] Lustre: 319096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11793.970146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11799.150506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11799.167974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11799.406855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11802.129727] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [11815.625244] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11821.919714] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11822.215300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11827.620916] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11827.898484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11835.375565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.377518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11835.422309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11835.440804] Lustre: Skipped 1 previous similar message [11836.879859] Lustre: server umount lustre-OST0000 complete [11843.554530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11843.558497] LustreError: Skipped 1 previous similar message [11843.561898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11843.569646] Lustre: Skipped 1 previous similar message [11843.573118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11845.610051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11845.612741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11845.638153] Lustre: Skipped 1 previous similar message [11848.072352] Lustre: server umount lustre-MDT0000 complete [11851.990829] LustreError: 317922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130449 with bad export cookie 15936770094695858645 [11852.000891] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11852.004743] LustreError: 317922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11852.458348] Lustre: server umount lustre-MDT0001 complete [11856.859964] Lustre: DEBUG MARKER: SKIP: conf-sanity test_111 skipping SLOW test 111 [11859.036737] Lustre: DEBUG MARKER: == conf-sanity test 112a: mount OST with no_create option ========================================================== 08:40:55 (1789130455) [11867.507434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11868.150163] LustreError: 321408:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11868.183690] LustreError: 321408:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11868.269113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11873.270548] LustreError: 321409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11875.532854] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [11878.373753] LustreError: 321408:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.514541] LustreError: 321409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.538866] LustreError: 321409:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11886.373539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11886.827267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11893.051930] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [11900.986815] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11907.685805] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11915.785942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11916.153737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11922.848684] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [11931.220243] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11939.176136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11939.404174] Lustre: lustre-OST0001: new disk, initializing [11939.418098] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11939.431909] Lustre: 324244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11939.589774] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11947.170308] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [11947.644766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11947.676766] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11947.808727] LustreError: 324626:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11947.832358] LustreError: 324626:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [11948.848529] LustreError: 324622:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11949.927348] LustreError: 324622:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0001: cannot cleanup orphans: rc = -1 [11949.940958] LustreError: 324622:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11951.972843] LustreError: 324626:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11951.988550] LustreError: 324626:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages [11956.152026] LustreError: 324626:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11956.178969] LustreError: 324626:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 7 previous similar messages [11958.406434] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11958.706055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11964.453246] LustreError: 324626:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 [11964.475604] LustreError: 324626:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) Skipped 15 previous similar messages [11964.905288] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9870e06c3800.ost_server_uuid 50 [11966.394287] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9870e06c3800.ost_server_uuid in FULL state after 0 sec [11972.556653] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9870e06c3800.ost_server_uuid 50 [11974.600144] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9870e06c3800.ost_server_uuid in FULL state after 0 sec [11977.026434] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11992.558534] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11992.559710] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11992.567515] LustreError: Skipped 1 previous similar message [11992.572284] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11992.581793] Lustre: Skipped 1 previous similar message [11992.601173] Lustre: Skipped 1 previous similar message [11997.881316] Lustre: server umount lustre-OST0001 complete [11999.204744] LustreError: 324627:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.236293] LustreError: 324627:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12007.392968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12007.398285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.422215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12007.435221] Lustre: Skipped 3 previous similar messages [12009.444155] LustreError: 323230:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.477619] LustreError: 323230:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12010.764949] Lustre: server umount lustre-OST0000 complete [12021.679363] Lustre: server umount lustre-MDT0000 complete [12026.559222] LustreError: 323252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130624 with bad export cookie 15936770094695859527 [12026.579743] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12026.588548] LustreError: 323252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12027.183771] Lustre: server umount lustre-MDT0001 complete [12039.448173] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [12042.104483] Key type lgssc unregistered [12042.342064] LNet: 326944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12042.357137] LNetError: 326944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12042.384873] LNet: Removed LNI 192.168.204.132@tcp [12043.268420] Key type .llcrypt unregistered [12043.271605] Key type ._llcrypt unregistered [12069.877575] Key type ._llcrypt registered [12069.879789] Key type .llcrypt registered [12069.978540] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [12071.281474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12071.303515] alg: No test for adler32 (adler32-zlib) [12072.504520] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [12072.829912] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [12074.535693] Key type lgssc registered [12075.793438] Lustre: Echo OBD driver; http://www.lustre.org/ [12083.290095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12083.321300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.843546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12090.111074] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12094.945754] LustreError: 328363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12094.967636] LustreError: 328363:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12098.892723] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12100.069310] LustreError: 328362:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12105.184513] LustreError: 328363:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12106.043886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12106.427690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12112.183845] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12122.411891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12122.831815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12129.060813] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12130.039711] LustreError: 330046:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12130.067397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [12136.430260] LustreError: 330046:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12136.454418] LustreError: 330046:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12136.938340] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12145.643251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12145.936911] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12151.311430] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [12152.500435] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12160.428794] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12167.324257] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [12167.606369] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12279.266471] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12279.274087] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12279.276844] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12279.289986] Lustre: Skipped 1 previous similar message [12279.308258] Lustre: Skipped 1 previous similar message [12283.751187] Lustre: server umount lustre-OST0001 complete [12284.397749] LustreError: 330045:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12284.429562] LustreError: 330045:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12294.113138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12294.122086] LustreError: Skipped 1 previous similar message [12294.124830] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12294.143489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12296.645509] Lustre: server umount lustre-OST0000 complete [12304.873866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12304.882405] Lustre: Skipped 1 previous similar message [12304.892753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12304.904742] Lustre: Skipped 2 previous similar messages [12307.504710] Lustre: server umount lustre-MDT0000 complete [12309.984884] LustreError: 332050:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12310.013680] LustreError: 332050:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12311.816163] LustreError: 328342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130909 with bad export cookie 6979256077505177500 [12311.817445] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12311.824577] LustreError: 328342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12312.355912] Lustre: server umount lustre-MDT0001 complete [12324.139522] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [12327.376676] Key type lgssc unregistered [12327.815730] LNet: 333313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12327.823834] LNetError: 333313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12327.841665] LNet: Removed LNI 192.168.204.132@tcp [12328.713591] Key type .llcrypt unregistered [12328.723628] Key type ._llcrypt unregistered [12377.810884] Key type ._llcrypt registered [12377.816541] Key type .llcrypt registered [12377.959735] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [12380.017302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12380.084244] alg: No test for adler32 (adler32-zlib) [12381.385983] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [12381.702322] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [12383.488267] Key type lgssc registered [12385.176609] Lustre: Echo OBD driver; http://www.lustre.org/ [12395.657940] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12395.673185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12397.098354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12402.068205] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12407.280438] LustreError: 335604:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12407.310112] LustreError: 335604:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12411.744000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12412.250231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12417.409270] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12420.555689] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12427.295186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12427.873264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12434.305056] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12437.098147] LustreError: 337092:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12442.223239] LustreError: 337833:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12442.234832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [12442.253712] LustreError: 337833:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12443.305795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12443.483596] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12448.769582] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:65) [12449.966978] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12459.967561] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12469.698702] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12484.066547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12484.077355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12484.087697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12484.587198] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12484.611880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12484.625572] Lustre: Skipped 2 previous similar messages [12489.649423] Lustre: server umount lustre-MDT0000 complete [12489.700310] LustreError: 336598:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12489.722573] LustreError: 336598:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12493.668687] LustreError: 337116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131091 with bad export cookie 15505725329620053691 [12493.671214] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12493.682934] LustreError: 337116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12494.115140] Lustre: server umount lustre-MDT0001 complete [12508.743142] Lustre: server umount lustre-OST0000 complete [12510.688951] Lustre: 334868:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789131092/real 1789131092] req@ffff983b35522580 x1876039805499776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789131108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12510.723843] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12510.731231] Lustre: Skipped 2 previous similar messages [12512.802101] Lustre: server umount lustre-OST0001 complete [12531.201182] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [12543.091548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12543.426271] LustreError: 340868:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12543.490192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12548.390836] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12553.704803] LustreError: 340868:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12553.726494] LustreError: 340868:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12558.028466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12558.573819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12563.289848] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12566.420694] Lustre: 342009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12574.279092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12574.854666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12582.618938] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12584.116187] LustreError: 342364:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12589.237798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [12591.861704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12597.263224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:97) [12599.321519] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12608.104445] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12611.952913] Lustre: 343880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12617.698717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12617.704234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12617.729865] Lustre: Skipped 1 previous similar message [12617.761129] Lustre: Skipped 3 previous similar messages [12622.823456] LustreError: 340868:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12622.837386] LustreError: 340868:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12623.056536] Lustre: server umount lustre-MDT0000 complete [12628.090187] LustreError: 340848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131226 with bad export cookie 15505725329620055392 [12628.096613] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12628.109816] LustreError: 340848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12629.100643] Lustre: server umount lustre-MDT0001 complete [12645.480523] Lustre: server umount lustre-OST0000 complete [12649.440330] Lustre: 334869:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789131231/real 1789131231] req@ffff983b41aa3840 x1876039805550080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789131247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12649.461148] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12649.475654] Lustre: Skipped 2 previous similar messages [12649.877248] Lustre: server umount lustre-OST0001 complete [12653.690558] Lustre: DEBUG MARKER: SKIP: conf-sanity test_114 skipping SLOW test 114 [12655.465415] Lustre: DEBUG MARKER: SKIP: conf-sanity test_115 skipping excluded test 115 [12657.773357] Lustre: DEBUG MARKER: == conf-sanity test 116: big size MDT support ============ 08:54:13 (1789131253) [12681.560188] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [12691.231169] loop0: detected capacity change from 0 to 1099511627776 [12692.028720] SGI XFS with ACLs, security attributes, quota, no debug enabled [12692.043171] *** VALIDATE xfs *** [12692.069555] XFS (loop0): Mounting V5 Filesystem [12692.104447] XFS (loop0): Ending clean mount [12693.600653] loop1: detected capacity change from 0 to 18691697672192 [12809.772562] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [12820.954900] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [12864.173281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12864.630933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12864.672022] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12864.839813] Lustre: lustre-MDT0000: new disk, initializing [12864.978442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12864.991167] Lustre: Skipped 1 previous similar message [12865.030454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12871.330456] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12883.408272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12883.618911] Lustre: 349019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12883.739931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12883.744883] Lustre: Skipped 1 previous similar message [12883.798962] Lustre: lustre-MDT0001: new disk, initializing [12883.922978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12883.937299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12889.032913] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12899.605246] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12906.170578] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12915.264364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12915.582118] Lustre: lustre-OST0000: new disk, initializing [12915.587090] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12915.600394] Lustre: 350988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12917.064159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12917.076592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12917.277364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12923.228444] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [12933.765509] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12939.076680] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12939.332666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12944.761882] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12945.005240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12947.937130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12947.942068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12947.951110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12947.965907] Lustre: Skipped 1 previous similar message [12953.085658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12953.095494] Lustre: Skipped 2 previous similar messages [12961.247137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12961.469553] Lustre: server umount lustre-OST0000 complete [12965.857564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12965.872520] LustreError: Skipped 1 previous similar message [12965.878312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12965.902453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12965.907065] Lustre: Skipped 2 previous similar messages [12971.885132] Lustre: server umount lustre-MDT0000 complete [12973.543368] LustreError: 349025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12973.568661] LustreError: 349025:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12975.680957] LustreError: 349011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131573 with bad export cookie 15505725329620056946 [12975.695547] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12975.699960] LustreError: 349011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12976.008186] Lustre: server umount lustre-MDT0001 complete [12983.863612] XFS (loop0): Unmounting Filesystem [12988.481420] Lustre: DEBUG MARKER: == conf-sanity test 117: lctl get_param return errors properly ========================================================== 08:59:44 (1789131584) [12995.572460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12996.083626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12996.088858] Lustre: Skipped 2 previous similar messages [13000.731294] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [13010.113926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13010.404841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13010.419084] Lustre: Skipped 2 previous similar messages [13015.258527] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [13023.671989] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13029.973444] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13037.232449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13044.521735] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [13051.537257] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13066.208208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13066.215924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13066.227327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13066.229570] Lustre: Skipped 2 previous similar messages [13070.357883] Lustre: server umount lustre-OST0000 complete [13080.368800] Lustre: server umount lustre-MDT0000 complete [13084.107155] LustreError: 353533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131682 with bad export cookie 15505725329620057828 [13084.108555] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13084.125952] LustreError: 353533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13084.491838] Lustre: server umount lustre-MDT0001 complete [13094.616992] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [13097.855781] Key type lgssc unregistered [13098.166542] LNet: 357196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13098.179055] LNetError: 357196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13098.213723] LNet: Removed LNI 192.168.204.132@tcp [13099.159441] Key type .llcrypt unregistered [13099.162758] Key type ._llcrypt unregistered [13130.870262] Key type ._llcrypt registered [13130.875295] Key type .llcrypt registered [13130.996360] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [13131.975120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13132.083671] alg: No test for adler32 (adler32-zlib) [13133.226896] Lustre: Lustre: Build Version: 2.17.57_140_g76d5844 [13133.481312] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [13135.200095] Key type lgssc registered [13136.722346] Lustre: Echo OBD driver; http://www.lustre.org/ [13144.817187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13144.850664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13146.265984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13151.704247] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [13156.835782] LustreError: 358764:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13156.859960] LustreError: 358764:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13160.763735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13161.208160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13165.904181] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [13174.694597] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13181.750432] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13190.008945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13190.490797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13197.119960] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [13206.506930] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13217.770806] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13217.780373] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13217.799064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13218.286202] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13218.299339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13218.305691] Lustre: Skipped 1 previous similar message [13219.662373] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [13222.269448] Lustre: server umount lustre-MDT0001 complete [13223.392136] LustreError: 359492:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13223.426626] LustreError: 359492:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13224.714881] LustreError: 358760:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13228.513650] LustreError: 358763:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13228.569621] LustreError: 358763:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13233.643445] LustreError: 358758:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13233.687785] LustreError: 358758:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13238.099402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13238.251989] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13238.264477] Lustre: Found index 1 for lustre-MDT0001, updating log [13238.320992] Lustre: 358752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13238.626317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13243.883293] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13244.461544] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [13246.476636] LustreError: 362290:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13246.488434] LustreError: 362290:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13246.514738] LustreError: 358756:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13246.642393] LustreError: 362299:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.132@tcp: cfg command failed: rc = -17 [13246.658504] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [13246.658504] [13259.234816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13259.236773] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13259.245085] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13263.599706] Lustre: server umount lustre-MDT0001 complete [13264.359313] LustreError: 359492:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13264.390595] LustreError: 359492:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13277.943340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13278.036483] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13278.044912] Lustre: Found index 1 for lustre-MDT0001, updating log [13278.117713] Lustre: 358753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13278.504169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13282.928736] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [13283.852276] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13285.896940] LustreError: 363418:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13285.911909] LustreError: 363418:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13285.927957] LustreError: 358756:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13285.936968] LustreError: 358756:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13286.025493] LustreError: 363424:0:(obd_config.c:761:class_setup()) osp: device 8 already setup: rc = -17 [13286.031832] LustreError: 363424:0:(obd_config.c:866:class_detach()) OBD device 8 still set up [13286.037240] LustreError: 363424:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.132@tcp: cfg command failed: rc = -17 [13286.049967] Lustre: cmd=cf003 0:lustre-MDT0001-osp-MDT0000 1:lustre-MDT0001_UUID 2:192.168.204.132@tcp [13286.049967] [13591.025627] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13591.030388] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13591.045233] Lustre: Skipped 1 previous similar message [13591.071780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13591.076377] Lustre: Skipped 3 previous similar messages [13595.006225] Lustre: server umount lustre-MDT0001 complete [13596.139313] LustreError: 358764:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13596.169994] LustreError: 358764:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [13608.864731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13608.955716] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13608.961459] Lustre: Found index 1 for lustre-MDT0001, updating log [13609.004251] Lustre: 358752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13609.297476] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [13614.486728] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [13614.589146] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13616.114962] LustreError: 371789:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0000 attach error -17 [13616.125827] LustreError: 371789:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0000: setup up failed: rc -17 [13616.137783] LustreError: 358756:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [13616.143360] LustreError: 358756:0:(mgc_request.c:614:do_requeue()) Skipped 1 previous similar message [13616.214752] LustreError: 371797:0:(lod_lov.c:208:lod_add_device()) lustre-MDT0001-osp-MDT0000: cannot connect to next dev lustre-MDT0001_UUID (-114) [13616.236276] LustreError: 371797:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.132@tcp: cfg command failed: rc = -114 [13616.250313] Lustre: cmd=cf014 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001_UUID 2:1 3:1 [13616.250313] [13937.121683] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13937.128844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13937.136440] Lustre: Skipped 2 previous similar messages [13937.151860] Lustre: Skipped 3 previous similar messages [13942.130317] Lustre: server umount lustre-MDT0000 complete [13942.248250] LustreError: 358760:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13942.263263] LustreError: 358760:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [13945.496035] LustreError: 361951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132543 with bad export cookie 13534665930460440954 [13945.501368] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13945.509801] LustreError: 361951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13947.366426] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13952.072693] Lustre: server umount lustre-MDT0001 complete [13956.298982] Lustre: server umount lustre-OST0000 complete [13965.088226] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [13974.759235] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [14011.456025] Lustre: DEBUG MARKER: == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 09:16:47 (1789132607) [14019.164045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14019.555931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14019.657887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14019.863418] Lustre: lustre-MDT0000: new disk, initializing [14019.973414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14020.013329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14025.264116] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [14036.313925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14036.451543] Lustre: 382397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14036.549576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14036.553633] Lustre: Skipped 1 previous similar message [14036.637267] Lustre: lustre-MDT0001: new disk, initializing [14036.714986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14036.751715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14036.762383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14041.604796] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [14051.392592] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14057.299187] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14064.471191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14064.817357] Lustre: lustre-OST0000: new disk, initializing [14064.822463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14064.831627] Lustre: 384367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14064.947363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14071.431717] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [14074.978248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14074.994072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14075.105331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14081.712983] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14090.723162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14090.745590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14090.755107] Lustre: Skipped 1 previous similar message [14090.775779] Lustre: Skipped 4 previous similar messages [14091.695244] Lustre: server umount lustre-MDT0000 complete [14095.266657] LustreError: 383194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132693 with bad export cookie 13534665930460442907 [14095.273110] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14095.273326] LustreError: 383206:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x2:0x0]: rc = -5 [14095.281519] LustreError: 383194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14095.598235] Lustre: server umount lustre-MDT0001 complete [14109.686501] Lustre: server umount lustre-OST0000 complete [14137.537390] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [14147.686631] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [14184.152973] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13900 sec ========= 09:19:39 (1789132779) [14186.433785] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:19:42 (1789132782) === [14189.885518] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:19:46 (1789132786) === [14233.477783] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [14237.319105] Key type lgssc unregistered [14237.620802] LNet: 390735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14237.635694] LNetError: 390735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14237.667266] LNet: Removed LNI 192.168.204.132@tcp [14238.555979] Key type .llcrypt unregistered [14238.559420] Key type ._llcrypt unregistered